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The purpose of this, study was to determine the 
the National Teacher, Examinations (NTE) Core Battery for 
tcher certification in Louisiana. This information was ; 
;sist in the recommendation and establishment of a 
on the N$E. The jury judgment approach was used. 
Panels of faculty member^ Louisiana institutions of 

higher education to validate the content of the tests in the Core C 
Battery and estimate the score that coulfi be Expected from a . 
minimally knowledgeable teacher candidate. The Content Review Panel 
found that 93 percent of the Professional Knowledge Test, 98 percent 
Of the MathematL^f Test, 97 percent of the Science Test, 100 percent 
of the Social Studies Test, 84 percent of- the Literature/Fine Arts 
Test,* 100 percent of the Reading Test, 98 percent £f the Listening 
Test, and 96 percent of the objective items on the -Writing Test were 
content appropriate. The Panel concluded tpat the NTE Core ^ Battery is 
a valid jneasutfe for the teacher education curricula in Louisiana. The 
, Knowledge Estimation Panel evaluated the difficulty and importance of 
each item bf each of the eight Core Battery Tests, then estimated the 
'proportion of items that the minimally knowledgeable teacher 
candidate would answer correctly. ( BW ) : 
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CHAPTER I 



PURPOSE OF THE STUDY i . v ;tV 

1^ study ..reported here was to determine the valid- 1 

ity of the. National Teacher, Ex^inatibns" .(NTE) Core .Battery, fox usje in. ; 
teacher certification in Louisiana. This information was produced fo 
assist in the recommendation and estabHshment of a required score on. 
the NTE . The report is presented to the Blue Ribbon Score-Committee^ fojr ' 
its cons ider at ion in recommending . a s.coxe to the State Superintendent of 
Education; and to the Super intendeht' f of his -deliberatibn in receiving 

.the Committee's recommendation and determining the score he will refer ■ 

• . ~ . ■ .' . ...<>■. ...... ' m '~ 

* ^to the State Board of Elementary aii^ 

; Louisiana law requires that any person ^applying . for initial cer- 

tification as a teacher (including those certified to teach in another^, 
state and. applying for certification in Louisiana) shall satisfactorily 

. pass an examination I" that includes English proficiency, pedagogical 
knowledge, and knowledge of his or fief atea of specialization (RyS. 
17 :7 (6) ) The law also requires that the State Superintendent of Educa- 
• t ion s choose th^ appropriate instrument, conduct research to validate the 
. applicability of the i^.trument J td^teacher education programs in Louisi-.; 
ana,' .and carry out research to determine the level at which the selected 
'test. : is satisfactorily completed. In conducting this research, the 
Super in tendeht shall meet with, and cons idei the s^u^ class- 
room teachers , representatives from "teacher organizations V deans of edu- 

1 Ration ; from Louis lana public colle&esj. attfd- ^ representa-^ 

S.tives:ofthe^ govern^ 



'■•£-3 



INTRODUCTION AND OVERVIEW : . • ^^ft 



. ' ■ . .... , • - • 2 

p / Educational Testing Service completed a validation study o the 
NTE Common Examinations andt^Area Examinations for the Louisiana Depact- 

:.; ' • - . . * : " :■. . _ ■.. : _ . t . 

ment N of Education in July , 1978.. ; FdlloT?ing this study; the State Super- ; 



intendent of Education a) determined which Components of the NTE were 

* - ■ . • ■ * 1 , * •* 

valid for use in Louisiana and b/^j|jBajbiisfaed'» scores required for cer-- 

tification in 18 teaching areas. The score required for .certification 

in each: area was a composite of the Weighted Common Examinations Total 

and: the appropriate Area Examination. 

' ""■ Since that time, Educational "Testing Service, the publisher *of the 

NTE, has replaced the Common, Examinations ^th the Core Battery. The 

first administration of the new Core Battery was in November, 1982U As 

the outline below illustrates , there are substantive differences Between 

the Common Examinations "considered in the 1978 validation study and the 

Core Battery that was the object of this- current Study ^ 

Common Examinations Core Battery , 



? 



one 195 -minute test • three 120 -minute modules 

components in Professional . • separate modules in General 

Education and General Knowledge -Knowledge, Communication 

^ith objective measures of typical Skills ^ and Professional 
teacher education training and Knowledge emphasizing the 

liberal arts basics . teacher as< a ^problem solver - 

and decision maker 

content in professional education, • content in professional educa- 
written English expression, social tion, social studies , . litera- 
studies, literature and the fine ture and the fine arts, 

arts,* Science, and mathematics science, mathematics, reading, 

•listening, and writing, 
(objective questions and essay) 

scores in four areas (Professional • unweighted scores from each of 

Education, Written English ;Expres- the three modules (General 

sion, Social Studies an^ Literature : Knowledge, Communication 

and Fine Arts, and Science and Skills, and Professional 

Mathematics) weighted to produce a Knowledge) . 
Weighted Common Examinations Total ■ \ ; 
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In the judgment of : the ' I^tiisiwavpepattmSiit of Education, the dif- 
ferences between the Common Examinations and Gore BSttery warranted* a 
cbmplet^ validation study of /the : Core' Battery; fheh-^partment was 
directed by the Superintendent of. Education to conduct a validation 
study to be- Submitted to the Superintendent and an appointed committee 



that would assess the validity of the Core Battery and jaasist ijj * the 
- establishment of a .qualifying score on the'NTE for teacher certified-, 
tidn. ' ' V r -: ; ' i -£ , -*T- - : 



SCORE' RECOMMENDATION PROCESS ^ 

Three major groups are involved in the validltio^ of the Cbr,e Bat- 
tery, arid the establishment :6f a new' NTE score required for teacher per- 
tification in Louisiana*. ; These are the Superintendent of Education, the 
.Blue Ribbon Score Co^ittee, and the Validation Teams. Figure i but- 

•• .' : ■- - ... • . •. • ■ • .' • ■■ • y ■ * 

lines the relationship among these groups. ■ , . , 




Stats Board of Etementary 
and -Secondary Education f 

(AOomoMi 

: t ' 
i- 

• s . , 

Statt SupirMondont 
/ - of Education 



Bus Hbton 
Score Curtynittet 



Team 

- (thk Tern 
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State Supe rintendent of Education 

The Superintendent is responsible for selecting a test to be tised 
in teacher certification and establishing the required score on that 
test. He -refers his decision to the State Board . of Elementary and Sec- 
ortdary Education. . .The Superintendent forms his : decision "after ; receiving 
'%' : '&:aX^ ;.tiie ^ec^m^endatiops of f the^j<BItif y §i,bt^a:: 

Score Committree about a required test score. 

.'*;•■■ V v^vj,^-;;^, U . : -^f -J- ^^^^^ 

Blue Ribbon ISeore Committee 

This committee was appointed by the Superintendent of Education to 
recbmmend a required score- on the NTE. The composition of the committee 
membership met the requirements of R.SV 17:7(6,) about those persons 
whose recommendations the Superintendent is to consider. The 'Blue Rib- 
boh Score Committee included teachers, principals , a local super int en- 

_ . * _" ______*_ : * 

dent of. Schools, representatives of ^ teachers' organizations, deans of 
schools^f Education, members of the governing* boards for higher educa- * 
tion, and persons who* represented the State Board of Elementary and Sec- 
oridary Education, business, labor, and civic groups.. A complete - listing 
pf the committee members is given in Appendix I -A. 

The Blue Ribbon Score Committee has two functions. The first is^ 
to recommend." tp the Superintendent which modules tall or some) of the 
Core Battery are . sufficiently valid for ujse in Louisiana. The second 
function is to recommend a minimal performance level required for 
teacher certification. :! In performing the latter function the committee 
must decide upon both required performance on the Core Battery and: the 
composition of the score or scores ' (weighted or unweighted, single * for 
the entire NTE or -expressed as a multi-component score): The validation 
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study reported. here is a major source of information for the 'Blue Ribbon 

"Score Committee in making its recommendation; 

- ~ \' 

Validation Team 

" .■'-»' 

The Validation Team members predominated by deans of colleges of 
education from * public arid private institutions of higher education i,n 
vlibuisiaira^ at , the request ;of , the ' Stat^^pepartment^- 

of Education i and ^these faculty members acted as judges in validating 
the Core Battery. The selection afid 'composition of the Validation Team 
is fully described in Chapter II, and its role is s discussed later in. 
this chapter and throughout the report. ; The members are listed in 
Appendix I-B. . The judgments of the Validation Team provided, the data 
for the Content Review arid Knowledge Estimation . components of this 
report . 

Technical Support ; 

Educational Testing .Service .conducted the 1978 NTE validation 

, ■ • * •' ^ ■- ■ ■. ■ # — 

study in £ contractual agreement with the Louisiana Department of Educa- 
tion. The Superintendent assigned ^responsibility for the 1983 studvjto 

.... ..V ... _ *; - -1-—. *. 

the Department of. Education, Office of Research arid Development;. Staff 

from this office was reponsible for designing and conducting the study, 
analyzing the restiits, ajtd writing the final report. Other Offices and 
Bureaus within the Department, particularly the Bureau of Highe.r Educa- 
tion and Teacher Certification, also served in managi^g^^^^^d^^^ 
providing consultation. Department of Education staff members involved 
in the study are listed in Appendix ' 



.- ; -Educational Testing Service maintained a Consulting rdl^ under 
contract the 1983 validation. This group provided procedural recom- 
mendations and the test items needed for the validation study, attended 
the Validation Team meetiri^p to ensure security of Educational Testing 
Service materials, and permitted special administrations of the Common 
Examinations in Louisiana in November, 1982 and March, 1983. 

$TUDY%ESI6N AND RATIONALE ' _ 

. It -should beT rioted th&t, while the' NTE is required, for certifica- 

»...:- - .. - : ... -'■ ...... . < ■ . 

*tion as a beginning teacher in Louisiana, it is not the sole requirement 
for certification. Therefore,; this study is concerned- only with the 
validation of the NTE. and does not address any other aspect of certifi- 
cation. This study was based upon the design that had. been developed by 
Educational Testing Service \and used in other States as well as in the 
1978 Louisiana validation ofvfche NTE. *The design and procedures are 
described fully in A Manual\ for Doing Content ; Validity Studies the 
National Teacher Examinations for Certification JJses (Educational Test-^ 
ing Services, undated). The preface to this manual argues legal and 
professional justification for the validation procedures. ■ 

The NTE Policy Council's guidelines for using the National 
. Teacher Examinations state:' ___ 
In 'keeping with the sentiment expressed in the 
decision rendered in the U.S. v. South Carolina /. . • 

(South ^Carolina 1977) case the Council requires _ .... 

that an, NTE us^ 
, thr—jrl-f^^ in that certifica- 



tion process. Further, the Council requires 
that a' validity study be conducted to determine 
the relationship between the . teacher training 
curricula of the state and the outline 'and spe- ; 
.cif ic. test questions for each test being consid- 
ered. In the process of conducting the validity 
study, if a minimum, score is to be established, 




it should \be "established in relation to some 
/ '* . criterion y such as the judgments of experts. 

This policy is consistent with guidelines for^ test use that 
have been promulgated by other, groups and agencies "(American; 
Psychological Association, the National Council for Measure- . 
ment in Education, and the Educational ^Testing Service, for 
example) as well as' with relevant decisions, rendered in the 
federal courts, (Educational Testing Service, undated; Pre- 'i , 
face) ■ • ' ' . ; ■ . 

^ Use of the NTE ^ - / 

Professional -opinion in the field of education is v not unanimous 
about the use of a test for teacher certification . T^te disagreements 
about the philosophy of this practice have been articulated by two major 
teachers 1 organizations. The American Federation of Teachers feels that 
V it is app^bporiate and desirable to test teacher candidates to ensure 
that they meet minimal standards, and "welcomes fair and valid teacher 
competency tests" (Shanker and Ward, 1982, p. 8), The National Educa- 
tion Association, however, endorses the rigorous evaluation of aspiring 
• teachers, buti feels that 'testing at the completion of a teacher educa- 
« tion program is ill-timed and ihappropriate (Hodgkins and McKefina, 
1981).- These authors argue that proficiency in the basic skills and 
liberal arts should be assessed before a student . has invest e^J^r^years" 
in a teacher education program^^They^h^venEwo^'further objection's: a) 4 
^tKSt^schoo Is of education and state licensing agencies are too far 
removed from the acquisition of skills and knowledge in specialty areas 
to assess these adequately, and b) that the paper and pencil tests -that 
are frequently used are generally too narrow V and inaccurate^ In the. 
judgment of those responsible for this study the last set of arguments 
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-.' does' jipt apply 'to evaluating tfie validity of the NTE for ^teachfer^ certif- 
. : ^ icatibii in * Louisiana. Fi'rst , the study does not ^address the point ; in a 
.^^d^rtt's educational ^ at Which the NT? is administered, Seqond, 

- the NTE* is considered a* measure of knowledge possessed by a teacher Qaii- 
didate j' not a measure of teaching performance; 1 ; Hence, the coHegV f ac-. 
, ;^ity .members respons ible for* preparing teachejr candidates' are appropri- 
ate ' -judges of the NTE V validity as a -measare of the teacher 'education 
" curribula. " The Louisi^a p legislaturi./h already 'decided th&t certain 
/ £area4 Ofvteachei? candidate knowledge ;sh4^ be assessed . ■ ' ' ^ 

v. /Begi^in'g- as early as^l^76 . fou.t; states required passing the NTE as 
' ,a condition ' for tfeaclier certification (Vlaanderen, 1982) . An additional 
23 states 'ti^ve' used the NTE for cert a fie at ion in specific ateas or for 
•• validating credits from* an unaccredited institution. ; : Eight; states cur-- 
\ rently use the NTE in teacher ^ceftijf icTation: Alabama v Arkansas, Lqtiisl- 
ana/ Mississippi^ South Carolina, Tennessee, ^irgl^ 

, |inia.- • < . _ '^^^ • .■ 

■* " ^ ^-Iur summary , ^/common practice -'and legal decisions ; to date 3 permit the" 
use of the -..NTE ifor teacher .certification when appropriate ' validation 
stu^ie^ have been^ employed/^^V ^ used , for the validation 

study reported here is felt to be appropriate becaus'e 'it is ,.. with minor. 
: additions , the oiierd^yeicspe^Jby publisher b£ the NTE, and. the test 
* has' - withstood legal ^ challenge- when , this^ validation procedure ' is 
' ■ ; • ; employed. The present study has added two. components 'that, jji the judg- 
: ! *ment of the researchers, contributed important abpiit teist 
content and "improved the : "quality; of the validation; these are discussed 
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briefly in the design sectietr-arid in the following chapters describing 
'the results. . . ' 

Design ' .. i\" ■: - 

The study ' employed v the jury judgment approach used in the 1978 NTE 

. •-' ' •■' . .. __ 
validation study;-; conducted by Educational Testing Service. In this 

method^ panels of Ifacuity members we're drawn from Louisiana insti^ 
^| higher"* education ^^a) validate' the * ^bhtent ( of tile tes£s: in tfe :©Sre \ 
Batitery and b) estimate the score that , could be expected from a^idini*^ 
mally knowledgeable teacher candidate. Educational Testing Service, had 
selected a jury judgment approach over other alternatives in 1978 
because none of .these could provide equally extensive and reliable data 
^within reasonable time and staffing constraints ' (Educational Testing 
Service, 1978, pp. 42-43) . The rejected 'alternatives--a detailed con- 
tent analysis o'f written documents or 'observation of NTE scores among a 
group of teacher candidates v indeperfdeiitly judged to possess a minimally 
acceptable amount of knowledge—appeared equally impracticable for the 
1983 study. * ^ , 

Figure 2 outlines the design for the study, faculty members who 
met qualifications concerning teaching areas and years of experience in 
Louisiana colleges and universities were nominated by deans of colleges 

of education. Panelists were then selected from this pool to form . a 

v * 
group that was representative of Louisiana institutions in three areas: 

« ••* • the number of teacher education majors graduating from the univer- 
sities; - 

:. • the proportion of public and private colleges, and universities; 



• the proportion of institutions with predominantly white and pre- 
dominantly black student enrollments ;; t ... _ • > 

The selection and composition of Validation Team panels are fully dis- 

-• •' . -' ■ v':V\- • • "'. vC-<" " ••••• . % . ' 

cussed in Chapter II. > f - 

: " ^i- • Val idat ion : Te.am members were^ a^s i^i^d to serve oh. veither. a Content 
Review Pine 1 or a Knowledge Estimatib^&aneri- and they . judg^ 
were • ■ : &ppv6prlai&0Jb the respective sub j ec,t*:vare£s;^ ••Wgdfe^ 

.■nominated. . ; V' :V;:v:;V ;: '^'-.v ■^■-■-■r. 

Each Content Review panelist made two major judgments about the 
Core Battery test to which he or she ^s / assigned. • 

• Is the emphasis given to topics within the test 1 the. same as the 
emphasis giveji to those topics in the Louisiana teacher education 

curricula with which the,, panelist is f Miliar? 

■ • • • . . 

• Would 90 percent or. more of the ^graduates/: from the teacher educa- 
tion programs in Louisiana with which the - panelist is familiar 
have had the* opportunity to learn the content included in each 
test item? ' ; ; - 

» ( «■» » ... a .'.■■'»• f .> 

the panelists 1 judgments about the congruence -between test topic 
and curriculum content determined how well the tests matched teacher 
education programs. Their judgments about the. appropriateness of items 
(whether students would have' had the opportunity to learn the required 
content) was combined with this to "evaluate the overall content validity 
of each test for L<iuisiana. Only* those items judged to be appropriate 
by the Content Review panelists were included in the^nalysis of the 
Knowledge Estimation results. \ In other wdf ds , -an itemyjudged to be 



Representative of Louisiana Colleges andillntversitief. 
(NiSber of Teacher Education Graduates, Pub1ic//Hvate; 
Hlnorlty/Hajortty Enrollnenta) : ■ 



COKTENT REVIEW PANELS 
(for bach teat) 



U emphasis given topica within 
etch, list (iiicr tpt ton cbngwent 
with eiphaaia given topica in 
curricula? 



ftould 901 orjwre - 
of teachfr. education 
graduates have an 
opportunity to learn, 
content of each test 
item? 



Match between domains in \ttt 
and Loutatona teacher education 

iorricotir ' ; ;• 



KNfflft*roGE ESTIMATION PANFti 
(for each test) 



Would minimally knowledgeable, 
teacher candidate find each 
itesTof little, medium, or 
considerable difficulty? 



la knowledge neaaured 
by each item essential, 
Important, or not Very 
important? 



Percentage of t test Items 
appropriate for Louisiana 
teacher (jandtdatea;: . 



What proportion of items at eacji 
coiblnatibtt.of ^difficulty/Importance 
(e.fr. r { easy/esientiai) would / 
minimally knowledgeable teacher 
! Candida te ansWct correctjy t 



Evaluation of content validity of 
each teat: arid of the modules they 
comprise for Louisiana* . 



Calculate ^omber of appropriate 
items minimally knowledgeable 
teacher candidate would; answer 
correctly to estimate required, 
scores on tests in Core Battery 
modules; : •<"■'.:■ • 



Examine NTE performance 
lti relation to ptbpoaed 
teat scores to estimate. 
Impact of score on supply/ 
of teachers, 



Louisiana determination of performance 
standards on appropriate tests and modules. 



FIGURE 2. : STjpf DESIGN 
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4S ' * ■» • . • _i. • ; v^ % r "%. t _ ' v *. 

•->;>;....' y, V Th^i Knowledg^ /Estimation panelists vjwere askfed* to det erfiin^rwl^ j; 
tkey considered to be the , inihipially' knowledgeable? 



ft' 



. person possessing the minimal amount of academic 'knqwied^e needed ^^^^ ■ - \ 
';: V V\.' complete, the college program required ; for certification in Louisiana -^ld/' v 

bi/teabh :: effebtiVelyv Using thiS minimally: knowledge^!?-- teach^i?;>caiidi> . ' r , 
■ ■ '• ■;»:>' .d^"b&:'as -a „framg~ pf reference, each ptee list then Jai^ " :r < 
' < vll '£ion&- abdutVtSLe: ^st .to whuLcli &4 oi^ had bfe^ff ■as^lgrifed: ; v > V ! 



Would the miniinally knowledgeable* teacher caHSl4ate ; f ind s , each it em 
easy, moderately difficult, or 'hard?* v ; ^ - •. ia • ;'V ^- ,/ 



; ; Is v the knowledge Measured ixn' each^item . es s^ent fal , ' ampbi;t aitt , • or 

l ' : r ' "' T;;: ; n0^\:y^ry important? v : v ^. ^ " ;: ; V ' .' -.v^'. -:-^;% . V^-t /.'vlSi 

* v '^:: What prqp^ttic^ o^ ; it 

v.; , v ;•• . 

v o V : ;>■ . , ficuity and vlmpo r t ahc e ^ easy/ ess etitial easy/impoi^^^ ^asy/ndtr* ^ 

■■ \ yery>im^ort k ant , ; : etc) ? ' P ' • ^•" •-I. ' * : ' - \- : „• 

' ^ The. ritSngs *of ■ 
"design^ :.\lh that earlier ' validation, judges had Jaeeri "a:sked J to 'ra^e o'ily '-• . 

« , 1 S' # * '■ 1 ; ^ "■ ■ ." '. T - ' ■■ o ;- : • .. " > • .-«!■ • * ■ i 
■ ■ \.'. _■ .. '. « «._■■ _ _ '. _ 4 ".--.a .. _■ _ > ; " _ ,t ;-. _ .,_ _.*_^f-.' - -» '- -V<ix. ;.: «_ . -. 

: : v ' It&n -«dff f icii tty i'* ; Jiic lading ; i#ktirigs v of , t i't^m ^^^orjfc ^ic.e • ^prov^eis anvaddi- " ° 
V '-I -. ; ^tipr^ar dimension ifo tfhe mefasu^emeht^Qf; validit^.^^^e; rationale for this v> : ^; . ; 
.? - -• change in th^; study design is. disfcuss^d^ ^ft Chapter:, iV. 1 "' 2 : ; . ^ ^ r^;i:v. 

The Knowledge Estimation >pane£i^ts 1 judgmerfi's 9 of the nilmber t of \ •.' ■* ■ 

• , - ;. ' ,". • •. • v.'^H . . --'f- _ ^ • '. - ' _ ■ v/v-- - -' 

' ' V items .in each of th^ niri r e dif ficuity, anS i&pdlrtince categories wereji^v'^: lv «,.; 

' ■ '"• ' ■■' V -v ; ■: .. ■ - ■: ■. 0 ; '* ,> **^- W' .1 • H 

> v * 4 ^ bined with theit judgtiep.ts-pf the ^percentage of items 'ii eicK c^egoiy S^;"*.^ , 

' ^ .minimally knowledgeable teacher canididate would arisWeif correct ly;.' °$h^se : v - - 

• • .'■ > % .; " *' *\ ' ' V" * .' '" ? '•.*r'^. ' ^H/f''*^ v."' 

r data ^wer;g further combined with the judgments about. iteiaa"ap^rb3^riateness 

furnished -by the Content Revieiw ;£aixeis to calculate the' number \of^ kppfo- * 

i*> ~i, ■■■•■*' ■• . " . ." , ... v >^'i-\v - "> ■>;./ ■ . .. ,. . , ; \ -v. t •. t ;y V ! .* • • ■: 

ERIC V : v - ■ - ' : , • 



13 



;/ griaite ' f^ems th^m teacher candidate, would answer 



correct l^><ih &he Gore ^Eattery Tests ; 
.-V^^^^v."'^ ; Tfie J 4 a.sir • el ement of; the validation study V involved a second addi- 
. ■:, tion ^Sc^^hi 1978 ^lidat ion procedures. This addition was the opportu- 
• nity for teacher candidates . who would He' eligible for certification by 
• September- IS ,' 1983^ anH who took the Core Battery in^Kbvember, 1982, or 
^a&hi, v: 1583^ to* also take the Common Examinationp^at no extra cost; The 



Common Examinations 'were no longer in use by/Educat iona 1 Testing Service 
v after the introduction of the Core Battery. 'Allowing" Louisiana ' teacher 
candidates to take both examinations/was a concession on the part of the 
testing company. The purpose*pf the double testing option was to avoid 
penalizing- any teacher j&krmidate who wished to apply for certification 
ise-£<^e^he-4retf^TE score was established in July, 1983. ■ t * ' . 

As the last step in Figure 2 shows, three sets of information were 
. provided, to assist in recommending and determining a required score on 
the NTE. These were: '- • ' ; 



Evaluation of the coi^tent validity of Core Battery Tests, judged 
by the Content Review Panels; 

A score that could be expected of the minimally knowledge 
teacher candidate, formed from the Judgments of the Content Review 
and Knowledge Estimation Panels; and ■. . ' 

Performance* 'information to be used in evaluating the - .ef fect of 
different qualifying scores on Louisiana teacher candidates 
attempting the Cbfe* Battery.' / v 
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. . j PANEL SELECTlp '■ ~ 

- g " . . . . ••, ' - 4fr : \ ■ '■■ ■ ■ - — - 

PANEL SELECTION PROCESS 

lite Validation Team member^pho formed the Content Review and . 
..' Knowledge Estimation Panels were - selected to i... represent equitably the 
Louisiana colleges and universities that have teacher education pro-, 
* 'grams, There were three objectives considered in the composition of 

these panels: a) designating members with tfcp required expert qualifi- 

' " / • • . ^ ■: *- -_ ' 

cations, b) representing teacher training programs -fairly by size and 

• . \- - . • - ■ ; ■ _. . , s..'_;;^._ 

public/private status of institutions, and c) representing colleges and 
universities with significant -black student, enrollments fairly. 

The first objective required that the college faculty members who 
■. were nominated as panelists have the ^desired length and kind of teaching 
•experience. Nominations were to be limited to faculty who had taught 
for two or more years in a Louisiana postsecondary institution, and ^who 
had taught a course' .within the last five years. The colleges and uni- 
vers it ies , were also requested to nominate faculty from the professional 
«: areas or fields th&t paralleled the ponteaj: of Gore. Battery tests, the 

fields for the tests withiir each module of the Core battery are outlined 
below. x yy 

.» i Professional Knowledge: theire was one test ^thto tfeis 

module. Faculty members were to u be nominated from the x 
academic fields of educat iona 1 psycho 1 ogy , instructional 
methodology, measurement and evaluation, and the sociol- 
/' ogy, organization; and administration of schools. 

~~" — f L ' ■ b ' ' - ' ' . • 

i General Knowledge: there were 'four tests in this module, 
i The areas for faculty nominations to each test were^ 

- : Science TestV biology, physical science, chemistry, 

astronomy, geology/ meteorology, and general science. ; 



Social Studies* Test: history, political science, 
- economics socio Tdgyi ^thropbibgy, and geography 7 . 

Literature/Fine Arts Test: English, musical and art.- . 
/" * Mathematics Test: mathematics and mathematics ': 

" — r • • ■ ' • ~' education i • • ~ — — < . 

• Communicat ion Skills: there were three tests in this 

module. The areas for faculty nominations to each test 
. Mtete: ,; . 

Reading test : English. . . * 

. Writing Test': English. ; . 
* >r Listening Test: speech and communications . 

• The second objective required that eacfi institution provide a pro 
portional number of 4 panelists for the study based -upon the number of 
teacher "education graduates in 1980-1981. institutions were asked to 
nominate a specific number of panelists determined by ttfe relative size 
* of their teacher education programs. For example, if^a university hqd 

graduated 10 percent of the teacher education' majors in 1980-1981, it 
was asked to nominate 10 percent of the panelists needed for the valida- 
tion study. \ ; v * 

The third objective was to ensure fair representation of universi* ' / 
tiers and colleges with significant black enrollments. The State Depart - 
ment of Education collected information about the proportion of ethnic 
minorities within the student body of each institution having a teacher 
education program. Each, institution was then categorized as - predomi- 
nant ly (51% or more) black or white. The number of panelists requested 
from each Was then determined in a manner to. ensure that the proportion • 
of nominations from predominantly black or white institutions matched 
the proportion of students from these' institutions . For example, if 25 
percent of the State's college and university students were enrolled in 



preddminaiitly black institutions, then 25 .percent of the panelists were 
to be nominated from these schools . It should be noted that this was 
^based upon the compos it loir^-of the studei 



ground of the faculty Members nominated. Predominantly white institu- 
tions; were free to "nominate nonwhite panelists, and vice versa. 

The President of each of the 22 institutions with teacher training 

programs was asked to designate a Campus Coordinator to manage the pane ^ 

_ _ _. • ^ ■ •' - J - -. - f 

nominations. Each Campus Coordinator was given a form listing the areas 

or fields from which nQminees were requested and the number of faculty 

members required for each v area. He or she was then asked to send the 

Bureau of Research, Office of Research and Development, a list of panel 

nominees with background' info.rmatibh on each. The Assistant Director of 

this Bureau was responsible '. for chairing the task force that conducted 

this study. ' The information included the number of years the nominee 

had taught in Louisiana, the major and minor fields of the person f s 

■ ■ ■ y i 

degree, membership in professional organizations, arid a list of courses 
taught in the last five years. It could not be >expected that all fac- 
ulty members would have taught all of the topics within the Core Battery 
tests (Science, fdr example, included" eight disciplines) . Thus, the 
faculty background data were used' to select panelists who would repre- 
sent a broad range of the required expertise. . 

PANEL NOMINATIONS ^ 

Table i shows the number of faculty members requested" and nomi- 
nated from each of the institutions. Three hundred seven nominations* 
were requested and 302 were received. Four universities nominated more 
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TABLE 1 



NUMBER OF FACULTY MEMBERS RE QUESTED (REQ) 
SHOWN BY INSTITUTION AND 



AND NOMINATED (NOM) ; 
AREA » 



Professional Math 
I Knowledge I i' ; 



Science Social ! 
I 1 Studies I 



Art ' Music ^ English' Speech I TOTAL 

i i i . 



REQ NOM REQ NOM REQ NOM 



NOM 



NOM '.REQ NOM I REQ NOM ' REQ NOM ^EQ NOM 



aptist Christi an 



catenary 



-4- 



illard 



rambling 



4 1 3 



'l 6 6 



24 



24 



ouisiana Colic fee 



_U_L 



SU-Batbri-Rottgc- 



3J. 



10 101 



31 



31 



SU - Shr e v e port 



1U IX 



7„ 



ouisiana Tech 



i ii 



byola 



_2_j_J_ 



cNeese 



23 



23 
21 



icholls 



-at 



21 



ortheast. Louisiana 



21 



22 



15 



13 



ur Lady Holy Crogg 


1 - 

X- i 






!_ 


1 ~ 

1— L_ 


=^4- 


— = H = — 

1— 


■ i 


i 

=* 


1 


— 1 


r— 


4 


t. Mary'i Dominican 


! * 


1 1 


i 


i 1 


1 1 


_ U 


- ,1 


— 1 — i 








4 


4 


outheastern University 


1 5 


5 1 


3 


4 1 3 


LI * 


_i-t 


1 M i 1 


6 








25 


29 


outhern-Baton Rouge 




* f 




4 I 4 


* 1 5" 




1 *1 1 


it 




4 




35 


35 



outhcrn-Ncv Orleans 



ulanc » 



riiversity, of Nev Orleans 



2 1 3 3 1 3 3 | 



T 



25 



20 



nlverslty South west LA 



\ i 1 i 

^ 1 1 10 10| 5 6 " 2S 



31 



otal Komlneee ■ 



j 55 55 1 36 164 36 36 I 36 43 1 12 ' <J[ U 



8 6 81| 34 29 | 307 302 
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faculty members than had been requested 3 and four submitted fewer hpmi'". 
nees than requested; ' Xavier University did not provide any nominees and 



^-<Ml<^-^a^-i-o±pat-a— ^ &^^-^1^4at4on--s^idy^ 



Within the subject areas of the tests, 90 percent or more of the 

number of nominations requested were received in all but two areas. , 

. - _-_ — -■ _.. . - ■ - - ■ ■> 

These were art, in which 75. percent of the requested nominat ions were 

received, and speech/communications , in which , 85 percent of the 

requested nominations were received. 

The - number of nominations requested and received was actually 

greater than the number of panelists required. The study design called 

for a total of 36 panelists in Professional Knowledge to form^ Content 

Review and knowledge Estimation Panels "composed of" 16 members and two 

alternates each. All of the other test areas were to have £4 panelists: 

* __.__«**" .__ _' - . ■ _ . _ ■ • _ _ _ -_ 

10 members and two alternates for both the Content Review and Knowledge 

Estimation Panels. All panelists, including the alternates, were to^ 
participate in * the validation' study. . The excess ^^ominations were m l 
requested to allow for a proper balance of expertise • among panelists, ' 
the possibility of faculty members ^withdrawing from a panel, and hohpar- 
ticipation because of unforeseeable schedule conflicts. Table 2 shows 
the size of the panels drawn for each test area and tfie number ' of pane- 
lists selected from each institution. A total of '205 panelists was 
selected from the 302 nominees.; Thirty-seven panelists were selected 
for Professional Knowledge, and 24 were selected for each of the ^fier ( 
test areas. The additional panelist in Professional Knowledge was nomi- 
nated by a predominantly black private university and was selected to 
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ALLOCATION OP FACULTY MEMBERS SELECTED, SHOWN BY IHStlttJTlOOMrfEST" 



toressional j Math i Science j Social j Literature j Listening (Reeding i Writing f TOTAL 
Knowledge Studies Fine/Arts . 1 : . ! 
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Northeast Louisiana 
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Representativeness of Panels 

Table ; 3 presents \the propoi^Bn of 198,0-1981,, Louisiana teacher, 
.education graduates and the * proportion of: faculty" from each university 
among the panelists who were nominated, selected;, >nd who actually j?ar-, 

ticipated in the study. There were only-two cases in which the percent, 

" ' . ; .v v ' v • • + •• .> . ' . - 

of faculty selected for the panels differed by more than half * a percent-*- 

age point (.005) from -the percentage of , the State s teacher education* 

graduates coming from that institution. These were Southern 'Universi- 

ty--Bat6n Rouges .which produced 11.5 percent of the teacher education 

graduates and .from which 10.7 percent df the panelists "were selected, 

and Southern Uiii varsity — New" Orleans, which produced 2 .5 percent of the 

State's teacher education graduates ; and from which ; 3*. 4. percent o.f the 

panelists Were selected. The percentages of "panelists 'participating Jpy 

institution differed somewhat from the percentages of ^panelists selected 

because some* selected faculty -members were not able * to- attend -the v-ali- 

dation study meetings, : * Ifoweyer , the; discrepancies' here were also small. 

- The greatest differrences' wer£-jf<>r .^rambling 

vided 8.3 percent of the select^dvpaneTists .and 6.8 -percent of the par- 

ticipati^g panelists, and Nichol'ls 'State University, with .7.3 .percent :of- 

the panelist selections and 7 . 9 ^percent of the participating panelist's . ' 

Tables 4 describes the paiiel selections in, terms o£ the. predominant,-, 

racial background of the student bodies in the participating institu- ' 



' TABLE 3 

PERCENTAGE OF GRADUATES AND FACULTY NOMINATED , 
~ "SELECTEDr AND - PARTICIPATING, BY INSTITUTION 













Ins t itut ions . - 


1980-1981 


v i 


Faculty 






Graduates 


Nominated 

— ■ 


Selected 


* . ' a 

Part ic ipat ing 


Baptist Christian . . 
Cptitetiatrv 

w CXI U CUUl JT 


0.6 


0.7 


' "v. 

0.5 


'. . 0;'s7-\^ ; 7-:l r 


0.8 


1.0 ' 


1.0 


1.0 


DiTlard ' v """" 


1.1 


1.0 


• 1.5 


1.6 


£5 t amb 1 in 2 


8.0 


7.9 


8.3 '" 


6.8 v 


/Louisiana College 


2.4 


1.7 


2.4 


2.6 v.-;' 


LSO-Batbri Rou^e 


10.0 


10 .3 


9.8 


9.9 


LSU-Sfireveport . : 


1.9 


2.3 


2.0 ' 


■ 2.1 . ' 


--Louisiana Tech- 


3.6 


3.6 ■ ... j 


3.4 ' 


3. 1 ; - 


Loyola 


-1.3- 


1.3 


1.5 


0.5 


McNeese * 


7.6 


7.6 • 


\_ 7.8 


8.4 


Nicholls . 


7.0 


7.0 


7.3' 


7.9 * 


Northeast Louisiana 


6.8 • 


7.3 


. 6.8 ". 


-■ . 6:8 


Northwestern State 


4.9« ; 


4.3 ... 


• 4.4" , 


■ •}. '■ 4.7 . , ' 


Our Lady Holy Cross 


1.2 


i;3 


1.0 


lift : 


vvSt.v Mary's Dominican 


. 1:3 


1.3 . 


. ,1.5 


1.6 


Southeastern University 


8.2 


9.6, • 


8.3 


JB.4 : ' . 


Southerns-Baton Rouge u 


11.5 


il.6 


* 10.7 


11".0 


Southern-New Orleans 


, 2.5* 


2.3 


: .3.4 


3.1. 


• IVlane v ..; .. • ' " _._ 


£ 0.8 


1.0 


1.0 


.- 1.0 , -.— 


.University of New Orleans' 


8.1 


6.6 


8.3 ' 


" -8.4 - V .;• 
9.4 ' 


University Southwest LA 


9.4 


10; 3, 


.9.3 . 


Xavier . ■ . * ' . ; 




0 i 0' 


0.0 


0.0 


TOTAL (N) b •. 7"". ■ 

- . ' < ■ ■ '• • • ' ' .. . : 
»- ■' * ' * . • 'v' ~"! ■ ■"!*« •" ■ 


99.9 


100.0. - 


. 100.2 - 


• '99.8 . 



^ 'includes nine qualified substitutes who replaced originally selected panelists. 
Percent s may not total 100. due to rottnding. , " N 



i±ppf% It also shoitfs the proportions of graduates and panelists from 
public private Dyeralli t the schools with a predomi- 



^jmtiy^i^fr - student enrol Itrr eii tr— graduatftd— 24^ii percent of the State's^ 



ieacher educatib percent of the selebted panelists' Qame 

'frafivte 22>.5 percent of the panelists who actu- 

al iy participated in the study. State -supported institutions produced 
89 .5 p^^^nt ?f ' the i986-19§l teacher education graduates. Some 90.1 ■ 
percent of the Selected panelists and 89 .9 percent _of the participating 
panelists were from these schools^; / 
■J.^:;h^yAl:yf^^s^ and years of experience of • the faculty members who were 
nominated and selected, and who participated in the study, are presented 
in Table! 5 . The prdportion of women faculty members participating 
* (31 . 9%) did not differ great ly from the proportion of women among the 
panelists nominated (34 . 8%) . Three faculty members were nominated who, 
' did not have the required two years of experience. These vp^rsons were 
not selected as panelists; The majority of the faculty nominated 
(74.1%) and participating (72.3%) had 10 or more years of teaching expe- 
- rience. " 

% V In summary, the panelists who wer| nominated and selected, and who , 
^ participated in the validation study, were representative of teacher 
educatibn programs iii Louisiana. The selection process produced no 
apparent bias in terms of the size of the teacher education program, or . 
v ^ t status of the universi- ■ 

ties represented. The selected faculty also matched the total group of 
: hbiniiieSs;; in sex distribution and years of experience. "-/■••\- 
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TABLE 4 

- • . . 

FACULTY MEMB ERS NOMINATED , SELECTED, AND PARTICIPATING 

FROM INSTITUTIONS CLASSIFIED BY TYPE 6F~ SSgBSTriAflD- 

PREDOMINANT RACIAL-COMPOSITION OF. -STUDENT BODY : — — - - -' " 

PERCENTAGES - 



y 






Facull 


;y Members . 


/ 


Institutional 
Class iticat ion ' 


1980-1981 
Graduates 
(Percent) 

\ 

— « : — : 


Requested 
(Percent) 


Nominated 
(Percent) 


Selected 
(Percent) 


Participating 
(Percent) 

* ^ : 






- — -■ 


— , — _ — — 




State Supported _ _ 
Predominantly Black 
Predominant ly White .' 


22.0 
■ 67.5 


21.8 
67.3 


21.9 
68.9 


22.4 
67.3 


' 'l 

20.9 
69.0 














m 

Private . 
■? Predominantly Black 
Predominantly White 


2.0 
8.4 


1 . 0 
. 8.8 


i n 
8.3 


i ? 

1.0 

8.8*f 


8.4 














All State Supported . 
All Private 


89.5 
10.4 


89.1 
10.8 


90.7 
9.3 


90.1 
9.8 


89.9 
•10 .0 














All Predominantly Black 
All Predominantly White 


24.0 
75.9 


23.8 
76.1 


22.8 
77.2 


23.4' 
76.5 

- ^ — 


22.5 
77.4 


Base Number 

» . « ■ 


4285 


307 


✓ 

'302 


205 


'191 
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FACULTY MEMBERS NOMINATED,, SELECTED, AND PARTICIPATING 
BY SEX AND YEARS OF EXPERIENCE TEACHING IN 
LOUISIANA: NUMBERS AND PERCENTAGES 



r 1 • 
Characteristic ; 

if 


V • ' . r ■ 
* Faculty ~- . . 


r " ' """ I ■ 

' - ' ; 1 . 

Nominated | Selected 

T • 

-. — : +- : 


r 1 

i ■ 

| Participating 


Sex * ^ ' 
Female 

Male ' 


1 • 

n % j n % 

• 1 ' ,, ■ 
165 34.8 j 66 32.2 
197 65.2 j 139 67.8 

1 


. 61 31.9 ; 

130 68.1 


Years of Experience- - 
Eess than 2 ." • 
2 to 5 

6 to ? ; 1 
10 pr more 
\ Unknown 


— +— — 

" n % | n % 

3 1. 0, | " 0 0 
35 11.6 | 28 13.7 
- 40 13.2 ■> | 25 12.2 * 
212 70.. 2 | 152: 74.1 
12 4.0 j 0 0 


n % 

0 0 
28 14. 7 ; 
25 ■ '13.1 >n 
138 72.3 
0 0 


Base Number 

_ - • ' ' ' i 


- — r — ' 

302 | 205 
1 


191 
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PANEL ASSIGNMENTS^ 



"\ The; selectfed Validation Team members were randomly .assigned to 
either a Content Review or a Knowledge Estimation Panel that w^s appro - 



priate to their area of expertise. i\ Panelists were notified of their - 
assignments by their Campus Coordinators and were encouraged to famili- 
arize themselves : with their institution's total offerings in the area of ^ 
the test^they were to evaluate. ^To help in this preparation, the: pane- •< 
lists , were also mailed rating task directions and instructions and teSt 
^content descriptions a month prior to the panel assemblies. 

„ • ■ .■■ . '\ ■ .• '. . a 

Content Review Pane l Ass ignmertte and Attendance 

Table 6 shows the number „of Content Review panelists selected to 
evaluate each test °and the college or university from which these pane- 
lists were nominated. The table also reports the number who actually 
attended the October 13, 1982 panel -assemblies. <; r; \ 

The Writing Test panelists evaluated * the objective -subtest on 
October 13, 1982, and essay subtest on February 17, 1983. The latter 
session was necessary because the panelists worked with samples of* 
actual essays from the November, 1982, Louisiana administration of the 
Core Battery. The. second form of the Listening Test was also not avail - 
able px October, and the Listening panelists were called back on Febru- 
ary 10 and 11, 1983, to evaluate that . form. * 

Content Review panelists were selected from* 20 of the 21 universi- * 
ties participating in the validation study, and were in attendance from ; 
all but one of these institutions. A total of 102 panelists was 
selected, and 94 (92%) attended. The smallest number of panelists * 



■ TABLE. fi 



NUMBER OF- 



hi 




TOTAL SELECTED: 102 
TOTAL ATTENDING: 94 



iSe ^eHstB,evaliiatea the objective and cbbsv subtesta in the Writing TeSt ^ / ; - ; : 
ER | C "« of listening T"est evaluated in separate sittings;; one panelist i\i * return for review of "«nd form.. * 

■™"3i ' ' i • , . '=■"•" V'i-' ': ; . . : V.:-...-. :.=•-• 
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attending* 1 for any test was 10 (Science and Literature/Fine Arts Tests); * 
Attendance was equal to or greater than ■ the number cal led for * in the * - 
design for each of the other tests and compared favorably with that in 
the 1978 validation study. - ■ ; 

Knowledge Estimation Panel Assignments arid Attendance : • : 

The numbers of /panelists selected for -and attending:, the Knowledge 
Estimation Panel assembly are shown in Table 7 by institutional affilia- 
tion. The Knowledge Estimation Panel - met* on October 14 and 15, 1982, 
The first day of the meeting consisted of training sessions to familiar- 
ize the participants with the validation taslts they would perform on the 
following day. Panelists evaluating the essay subtest of the Writing 
Test /and tjie second form of the Listening Test were reconvene! in Febru- 
ary, 1983, along with Content Review Panelists for these jtests. 

Twenty of .the universities taking part in the validation study 
contributed faculty members tp the Knowledge Estimation Panel selection, 
and panelists attended from each of these schools. A total of 103 pane- 
lists was selected. Ninety- seven of them (94%) attended and 93 (90%) 
provided data that were used in the study. Four panelists were disqual- 
ified because they inadvertantly failed to meet the procedural require- 
ments of the study. These panelists do not appear in the Knowledge 
Estimation results, The smallest attendance was for the essay subtest .4- 
of the Writing Tgst* (seven panelists attended) and the Listening Test 
(nine reviewed tfie* first form, seven reviewed the second). These hum- 
bers were below those called for in the study design but were considered 
sufficiently .large to allow validation analyses of these tlsts with the 
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Knowledge Estimation PanSl. In the case of the Writing* essay subtest^ 
li Content Review panelists also performed the Knowledge Estimation rat- 
ing tasks. ' This was due to the nature of the essay subtest, which dif- 
fered in form from the other Gore Battery Tests > and for which formal 

Content Review ratings would not have been meaningful*. The greater of 

■ *> * 

number of ratings thus increased the stability of the data. * Attendance 
was" above the number desired for all other tests. 

HALF-PANELS 

After the October rating sessions, each attending panelist was 
assigned to a half -panel. These half-panels^ wer§ formed to measure the 
consistency of responses among judges on the Content Review and Knowl- 
edge Estimation tasks. Half -panels never met as a group or performed 
any duties separately, * °\ " 

^Tie two half-panels for each test were constituted to be as simi- 
lar as possible to one another With regard to average tenure, represen- 
tation of predominant ly black universities, and the departments in which 
the judges taught. This was done to form half -pane Is that would allow a 
comparison of ratings by the two groups that would not be ' expected to 
differ because of these factors . In measuring consistency for each 
test, data from the two half-panels were treated as if the. groups had 
given independent ratings on the Content Review and Knowledge Estimation 
tasks. The analyses of the half -panel data in Chapters III and IV are 
measures of intragrdup (half-panel) reliability, i.e. , a measure of tjtie 
consistency of the ratings . Rating tasks on which the half-panel rat* 
ings do not agree consistently may indicate a need for caution in inter- 
preting the 1 results of that analysis. ; 



Selection and Composition of Half-Panels 

Judges were allocated to half-panels bit the basis of three . cri- 
teria: a) the number ^of years each had taught in Louisiana, bj, tl^ft 



racial composition of the student body at the university in whichr the 
judge taught, and c) the department in which the judge taught. The 
half -panels that resulted from this *s elect ion process were deliberately 

■ * 

similar to each other. Table 8 compares the half -panels for each test 
on size, .average tenure of members , and the proportion of panelists from 
predominantly black institutions. 

In can be noted that the average tenure of the two half-panels for 
any test did not differ by more than 2.8 years. The mean difference in 
tenure between the half-panels for a test was 0.9 years. Similarly, the 

: _ '_ __ _ ■ > • 

half -panels were constructed so that each included judges from predomi- 
nantly black institutions when possible. Only three of the sixteen 
half-panels did hot/ include one or more judges from a predominantly 
bla^lc institution. These three exceptions vwere instances in which 
half -pane Is had been constructed from Panels that-: contained only one' 
jud^e from a predominantly black institution, and that judge was arbi- 
trarily placed in the other half-panel - of the pair. 

Finally, the department in which each judge taught ; was , considered 
when half -panels were selected. This vwas especially important for the 
-Social Science and Science Panels, ; ;in which a variety of disciplines was 
represented. The assignments were based upon distributing judges from 
the same disciplines j indicated by the university departments in which : 
they taught, to different half -panels within each test. 
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ROBBER M CHARACTERISTICS OP JljOGES, ON. CONTENT " '•' " ;'K\? t 

' ' * 'REVIEV AHi) WEDGE EStiHAtiON , ; ' , . 

' : RALf-PtolELSj BY TEST' ' „ ; ' ,,V - ' 



■ , . ff ' • 


i 

Nuiber 


| CONTENT 
Panel 1, 

' Average! ' Met 
Years oft For Black 
Tenure Institution 


REVIEW ' 

. 1 .Panel 1 
1 Average! > Number. 
' • | Years of 1 f dr. Black 
Number, Tenure [Institution 


Number 


"_ 'KNOWLEDGE E 
Panel 1 | ' . 
..Average j_Nui°er. : 
1 Years of 1 for, Black 
Tenure Institution 


STlMlO 
Number 


JJaneU 
Average. 
Years of 
Tentite 

t 


-Nunber 
fer Black 
Institution. 


Professional Education 


9 . 


' 18.0 




6;. 


" 

; "17,1 




8 ' 


16,0 


- 

2 


8 ^ 


1 . 
16.0 


; X ... 


Mathematics 

t 






1. 


5 

0 


'■til, 


h— 

' ' ; l 


5 


14,8 


0 ; 


•":'5 ; 


•15,0 


1 


... — ? 
Science 




15.8 


1 

• 


5 . 


J : 
17.0 | 1 

« 

1 M, 0 - ■ 


1 

.5 | 15.4 




; 6 


■■ 


i ' Ik 


Social Studies ' 




' % ii.2 


* 

0 


6 


- — n 

12.0 


1 t i 


< 

6* 


' 13,8 


1 


• 6 , 


:. 

';-12;0 

: 


i' 

— 


fctterit«re°/Fine Arts 

• ^£ 




10.4 

i 4 




5 .• 


11.8- 


; i 

* » 


i 


16.0 


' '!" ' • 


5 


13.2 


2 

^ ^ 


R ding ""'■*• 


■ . ; if 
. '6 


: 15.; 


i 


'6 


4 15.0 


i 


'6 


14.8 ■ 




— 1 

6 , 
• 


- 




Listening 8 (Fora A) 
(Font B) 


« 

5 


16.6 


' .i. 

2 


5 
5 


14.8 

i: 

17.4 


r 
, i 


5 

4 


h— i 

16.0' 
18.5 


. ; ■ 
• i 

1 


: 

V ' 


16.3 

1 

17.3 



■ 

0 

0 * 


Writing $ (Objective) 






r 




:- 12.8 


; 

i 


— 


h— H 

16'.0 




— 1 

' 5 ' 

— H 


15.6 . 


i 


Mting b '(Essay) 














— 1 

9 


13.4 


2 


9 


13.2 | 


2 • 

r 
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8 listening Test .tewieige Esciiation' inchdes separate 'half-panels for forms A (upper line) apd Forn B (lower lliie) of that teste this procedure 
r 9 acconosodates dlfferejaces in attendance at rating sessions for the two forms* Z 1 - ; ly ^ 

There ws no Content teview Panel for the gasay subtest of the Writing Test. . v 
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It has been , rioted that the Knowledge Estimation Panelists for the 
Listening Test evaluated Form A in October, 1982, and Fbrm : B in Febru- 
ary, 1983. Only seven of the original nine panelists were able, to 
return for the second rating sel^iori. Because of this, it was necessary 
to construct two separate sets of Knowledge Estimation half -panels, one 
for each Form of the Listening Test. Those ..p&I^Ssts who attended both 
rating sessions are included in both sets of half -panels. Those who did 
not attend the second session are excluded from the half-panels for Form 
B. This procedure was followed to prevent an imbalance in half -panel 
assignments. * . ' 

The essay subtest of the Writing Test was not included in the Cony 
tent Review since the subtest did not include discrete items, silence 
half -panels were constructed only for the Knowledge Estimation tasks. 



SUMMARY ^ 

V : ' ... - '. \- : ,,. i ; • ■ : . _ ' ' X_ 

The information presented in this chapter demonstrates that the 

_ £ ■ . : _ • - &z ' J f*& 

Know 1 edge*vvEs t imat ion dnd Content Review, panelists -who evaluated the NTE 
Core Battery provided a comprehensive representation of teacher educa- 
tion programs; in Louisiana. All but one of the State's universities 
with .such programs were proportionally, represented among the panelists 
These faculty members , as a ; group, reflected the balance of public or 
:pSivate, and ^predominantly black or white-, universities corit&iried witHiii 
Louisiana. They also represented a wide range >of disciplines through 
the departments in which, they taught-, and possessed a broad range of 
teacher education experience. The. panelists, in summary, were>: highly 
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qualified- and equitably selected judges of , the * validity of the testes 

they r eviewed I ' ( 
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CHAPTER III 
CONTENT REVIEW RESULTS 



INTRODUCTION 

The Content Review phase of the validation study involved four 
types of judgments' kbout the Core Battery 'tests : a) the content appro- 
priateness of the test items for students who have .completed teacher 
education programs in Louisiana, b) cQmprehensiveness, i.e. , the extent 
to which the tests included major content topics, c) the match between 
the emphasis given content topics in the tests and in teacher education 
programs , and d) the overall similarity between the content of each Core 
Battery test ind that of Louisiana teacher education programs . 

The composition of the Content Review Panel is discussed in Chap- 

: < ■ ' . __ • ; _ • _ __ i _ ___ __ . .' . _ 

ter II/ The panelists met on October 13 , 1982, at which time their task 
descriptions and instructions were reviewed in detail* and the item and 
test validation ratings were carried out. An overview of the tasks car- 
ried out by tTte Content Review panelists tv is included in Appendix III-A. 
Of the 102 Content 'Review panelists selected, 94 attended the Octpber 
panel assemblies . The Listening test panelists Reconvened in February,' 
1983 i to * evaluate the second, form of that test, and* the Writing Test ■ 
panelists were also, called back in February to'serve on the Knowledge 
Estimation Panel 'for the essay subtest. f : ' 



ss of items ^ > * 
. : %^;^dge^fl^ed tb^Qtie^tibft Review Form (Appendix III-B) ^td evaluate 
thS^pfit^n^a^ each item in the test to which they had 

*• * ' j/;^" ' ' ■ i" - . • " ""V* 



erjc ^ so 



been assigned. Art item 'was to be rated as content appropriate if the, 
panelist judged that 90 percent or more of the students in the .Louisiana 
teacher education programs with which he or she was familiar would have 



had the opportunity to %eaxn the knowledge required to answer the ques- 

» , . .... . \ . ■ . . _;. 4 ■ ' . , . . . r * . 

tion. This learning could have occurred during the student s teacher 

'-'-.■'"'(' ■*-■ ■ ■ ■ ■ - - - v 

education coujse of' study or as part of coursework prerequisite to the 

: ' \'_ ._ . •_ ? _ 1 A _ _ ^_ 

t-eacher education program. Judges were instructed to rate each item as 
either- content /appropriate or inappropriate unless they had no basis 
whatsoever for making the judgment. In the latter case they could 
assign a ,f Do Not Know" rating; ; 

The test items were first analyzed' individually. Some judges 
selected' the ff Do Not Know" option / so not all questions were classified 
by ail judges. e The "Bo Not Know" responses were treated as neither 4 
appropriate nor inappropriate, and w6re thus not included in the data 
analysis. The exc lus ion of the "Do Not Know" responses was based on the 
assumption that, if these judges had had a basis on which to rate' the 
items, their ratings would have Been distributed in the same way. as 

jse of the other judges who- rated the items as appropriate or inappro- \ 

i, ■■ ^^v. " • ••'•i'V- *'-' < ' ■ ' i ' 

priate. This procedure had been followed by Educational Testing Service 

in the 1978 validation study. It was also designed to exclude any* items 

that were rated (as eithei appropriate or inappropriate ) ; ;l?y fewer than 

■; three, judges, on a panel: or half -panel. Npne of the items from any test 

required exclusion on this, basis; "all were rated by three or more judges 

oh both half-panels . * 

• • .-».*■.*.■ - '. r ii; " ... • ' -. ' ■ ' /" 

\ Each item .was then classif ied as^eitHer" appropriate or inappropri- 

ate in the judgment of • the total panel evaluating the test in which it 



appeared. An -item was classified , as content appropriate if 5i percent 

or morB of the judges gave it this rating.. This "procedure was identical 

. ___ • _ _ _ - ._ k _'• . ° . <■ 

to that^used in the 1978 validation study; in actual practice the cri- 
terion^ was somewhat 0 higher than 51 percent. In the largest panel (Pro- 
fessional Knowledge Test) the majority was nine of 17 panelists, or S3 
percent. The majority was six of 10 in the -smallest pfitell, or 60 per- 
cent.. • ; -J " / ' * ■ 

■ ' - : . • ■■■■■ . v.-x;.' ■ ■■ ■■■ • - ■ : . -' ■ • . 

Table 9 displays the number and percent of items, rated as content 
appropriate for bath forms of each tlorei/ Battery test. The number and 
percentage of items given for the Writing Test exclude the essay 'sub- 
test. That subtest consisted of a topic on which the examinee^ was 
instructed to write an essay and did not inciucte discrete items that 
could be evaluated for their, content « appropriateness . 

Across tests , the percent of items judged content appropriate 
ranged from 100 percent (Reading and Social Studies Tests) to 84 percent 
(Lit erature/Fine\' Arts. Test) . It should be noted, that only those items 
rated as content, ^appropriate by the majority (51% or more) of the judges 
could contribute to the recommended performance standard on the. Core 
Battery.,'' «• "''[;..' . .\; 

When "the tests were considered as modules, the percent qf items 
judged appropriate ranged from 92.3 for Form A of Professional Knowledge 
to 98.3 "for Form B of Communication Skills. It is important, to consider 
the item appropriateness of the modules because Core battery scores arfe 
reported for the three modules rather than for their component tests. 




.;' . TABLE 9 ■ — 

} NUMBER AND PERCENT, OF ITEMS RATED APPROPRIATE FOR 
EACH TEST 'FORM ANti MODULE AND AVERAGE PERCENT 
* APPROPRIATE FOR TEST FORMS/ AND MODULES 



Ptbfessioiiai Knowledge 
test /Module 


FORM A - 


FORM B 


_ 1 _ _ ' _ 0% : *- A 

■ AVERAGE 


Ntimber Percent 
of of 
r- Items It§ms 


Number Percent 

of of 
Items Items 


• Percent 
of : 
Items 


96. 92.3 


96 94.1 


' 93.2 


. Mathefetics Test 
■ ' Science Test 
1 Social Studies Test 
.Literature/ Fine* 
'Arts .TeSt 

o . ' 4., 

■ ' ■ - .- ■ " ■"- 

General Knowledge 
Module v - ';■ 


24 -96.0 
; 29 96.7 

30 100.0 j 

31 88.6 , 

..... v-.v^y* -'if' ' 

114 95.0 


25 100.0 
29 96.7 
' 30 100.0 ; , 

• .27 79.4 
111 93.3 


.' .98.0 .'. \; - 

96:7 v :■',.;./•-■'%■:. 

100.0 : .' 

,. 84.0 . ' ' 

■>'■■ . • • . ' ■ . 

.••■•/.•« 94J : 


Reading Test 
Listening Test 
Writing Test, ' x 
(Objective) 

eotnmunf cation Skills 
* "Module 


30 100.0 ; 
39 97.5 

42 93.3 
111. ' 96 V .5 


30 100.0 

. 39 ;: ',.,97.5: 

44 " 97.8 
113 , 98.3 


* ioo.o * 
V 97.5 : ■ '* ■ 

' 95.6 
r 97.4 



Suin of the percentages for Form A and Form B divided by 2; 
Excludes essay subtest. :/ v „; 



: ■ 39 

Comprehensiveness of Tests ^ 

A - ' ~ • - • ■ - . :'V -'. _ - , - -: 

Judges were, asked to identify aiiy major .topics that were inbliided 

.■ . ■• .•'.*••■*■• - *.rs * ■ 

in the teacher education sequences of their institutions but r did not 
appear in the content description of each test (see Appendix III -C). 
This was done in order, to determine the extent to which the tests omit- 
ted important content. Forty-three of th,e 94. Content Revi&w panelists 
completed a Test Content Omission Form. The results are summarized in 
Table 10. ; - . • . . , 

• Of the 43 panelists' returning the form, 36 cited specific topics 
that were* included in their institutions' teacher education programs but 
omitted *in the Core Battery test content descriptions. The remaining 13 
panelists; who returned forms either indicated that no topics^had been 
omitted or made general comment^ that. wer<e not . contend specif ib/? about 
4&-test..:.or the -teacher education curriculu^^ i^e number* of omitted 



I topics c it ^d ragged from rone (Reading Test ) to 11 (Writing Test). , Only , 
seven topics were "Sited by. two or more judges . Two. of these ; topics were 
cited- for the Listening Test: analysis and synthesis of oral, cqmmunicar 
tiori, and stimulus -jresponse questions for oral communicatioh. TWo judges 
noted that the Literature/Fine Arts Test did . hot include items on the 



recognition of names and works of famous artists. In the Mathematics 
Test, three, panelists noted the Emission *of probability, , and two cited 
statistics and algebraic problems as omitted topics . Finally, in the 
Writing Test (.Objective) /the^topic \of spelling of frequently confused 
words was reported to be omitted by two judges. 
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7 TABLE 10 



_ ; TOPICSJW OTE_ ^ - _ ' . 

AS NOT INCLUDED IN THE TEST CONTENT DESCRIPTION OF THE NTE CORE. BATTERY 



AO 



Number of Judges 
* ^turning Totm- l 



: % .Topic*; . 



Frequency 
of Mention 



>£ essional Knowledge 



6(3) c 



Developmental processes 



:hematics i£ 



:8(D 



iiit- Studies:; 



; 2 



4(2); 



erature/Fine ; Arts 

iding 
itening 



5(2) L 



3(2)^ 

- v 

7(l) 1 



ting 



8(2) c 



Multicultural characteristics., of ■ 
- contemporary school settings i 
Pu^e psyc1ib^o^-{^ 

Specific educational theorists. & theories 

^Probability^ ' v 5 ^ : t' v . 
Statistics :j5cV J . 7 '-/-r ~ ' V 

Algebraic problems " 

..Trigonometry ; ."; .'7%, " ~ 
Set theory J . .." '~ 
Consumer oriented problems} "'"-'zr 

-Inf ormal : geomplq^ 



1 



Properties of real numbers ... i 

~--v : - ■ _ V . ; -V.J *•■'>-! 

Motion -'>.:•■ £* f . - • 

Work'* ■• : \~~ ■ /.:]'](• • 

S imple machines _ ; * > ^ 
Phyfical^hergy* v^^ : ^ 

Laboratory techniques! v ' 

'Contemporary American, government 

Political issues; ^nd; topics ; ^ ; 
' Constitutional and legal topics-,- 

The Middle Ages . T..— 

^Euibpean^wlture^:^ - • ^ 

Early civilizations ^ '1 ,^7* 
-Syplutionary fbrces ' ^ 



;3 
2 
'2 
1 

lr 

1 

1 

TT 



"Recognition" of -names and works of famous people 
^<Ld^el opment in the arts ; ; / 

as tied to content areas £~ '.*'<■■■ 



iomprehension of sequencing of evejits ; 



Analysis and: synthesis of oral co mmun ication 
Stimulus-response quastipns f or communication 

'V/' •. " : :! ' ; ; : &_V' : 
Rhetorical, situations v^A\ • f 3* ^ 
Modes of 'persuasion j.'y. l""^" 1 ' • ^>>"" 
Psychology of iistehing vi.- hearing" i- • > ^ 
Message strategies;..^ • ->.: ' t \\ t V. 

Minority cultures 

Following directions tt y , " 

Evaluation of the message ■ 



Spelling 'of frequently confused works, 

<[ive; i h tpVtooi threw-through) 
Standard vs. non-standard English verb forms, 
Transformational grammar * ' 

tinguistic principles - J • 

Rhetorical principles ; , / ■ '* 

Comma splices c ^ 

Principal- par ta of verbs : 

Library techniques Y : ; 

Research papers "i, 

Composition • 



; 1 
,1 

- XV 

; 1 
1 

1 . 

1 V 

: 1 

i ■; 

-^2 

. :i *'. 

3 

I 

,■ t '< 
i 
i 



2 
1 

i 
l 
i 

i 
1 

i 
l 

i 



to Appendix: III-Ci Test Content Description, .for a listing of topics included in the Core Battery Tests. 

r.o/ judges within She total shown who returned form but indicated that no topics" were omitted and /or wrote 
nents about the test. 61 



Relativ e Emphasi s of Tests ggd C urricula 

The judges were asked to evaluate the match between the emphasis 
given content topics in the<Gore Battery rtests aid, in Louisiana teacher 
education programs. Panelists were ' given a Te§£ Content^ Description 
(Appendix III -G) showing the percentage of the test that was devoted to 
each content topic. They were then asked to judge whether that topic 
received the same, more, or less emphasis in Louisiana ^eacBer education 
_ xnrrigul^jhafi in the_test . Pane lists were told to disregard differ- 
ences of 5 percent or less. This restriction was imposed to encourage 
judges £6 focus v on only those meaningful differences that would indicate 
a true divergence between the tests and the curricula; Table 11 pres- 
ents the frequencies of the three relative emphasis ratings for each 

.test. , ■ / •/ ' 

The "column titled "Proportion of Same" in Table 11 shows the dif 7 

> ; - • • ' .. '• t ~ • - ... _____ . ____ '__ 

f erences in emphasis for each test. The value* given each test in this 

— '■* 

* column was computed by dividing the number of 7sa^e" responses given by 

■ , 1- , : , "■ \:V. ' : - - : v-v v - - ■ _ J 

the- panelists by the total number of responses ("same" plus "more" plus 
"less"):. :^or # ixS Same" .value for the Profeg- 

^ sional Knowi^ge>. Test was : ; • i ; ' % 1 

• ' ••' " . .' : • =. \ - . ' t< • .•' ';' f - • '.. " 
- 58 (ntunber rating emphasis ais ^am^ 1 ^' - .57 " 

102 (number -rating stone , " "more^^less R ) , : 

When this proportion was .50 or greater it meant that, for the test as a 
whole,' half or more of thi panelists Judged that tgie emphasis given dif- 
^- f erent topics in teacher education* programs in Louisiana was the same as 
the emphasis that topic received in the test. ; 



• TABLE U ..; 

FREQUENCIES DP JUDOCENTS ABOUT RELATIVE EKPHASIS 
IN TESTS AND CtMlCULtiM 



42 



Topic 



Percentage 
of Teat 



/ Number Rating Emphasis ^ 
Given Topic in,, Curriculum: 
-Less [ •"- Smo* [ Morez 



Proportion 
of Same 0 



_Degree of^ 
Difference 



)feasional Knowledge 
ist /Module 



1 
2 
3 
4 
5 
6 

total 



24 
25 
i 12 
9 
11 
14 
100 



2 
3 
5 
7 

: 6 
6 
19 



11 

id 

10 
7 
10 
10 
58 



I -10 | 1 



4 
4 
2 
3 
1 

i 

15 



1 



tethematics test 



Science Test 



iocial Studies test 

' '/' " r " h 

.iterature/Fine Arts Test 

_era4-Kn^ledge Module^ _ _ 
leading Test 

.' * 

listening Test 

Writing Test (Objective) 
omm|caticm Skill Module 



1 
2 
3 
4 
5 

• 6. 
Total 

1 
2 
3 
4 
5 
6 
7 

a 

9 

total 

1 
2 
3 
4 

•i Totals 

1 
2 
3 

Total 



1 
2 

: 3 
.Total 

1 

■ ~m 

. -^4 * 
total 

1 
2 

Total 



2<T 
16 
-24 
16 
12 
12 
100 

11 

ii 
n 
it 
ii 
ii 
ii 
ii 
i^ 

99 

25' 
25 

100 

29 
54 

-17- 
.100 



50 

. 35 
15 

. w 

30 
18 
15 
100 

55 

45 



2 
2 
1 
2 
4 
_4 
;15 

3 
I 
2 
2 
1 
0 
2 
3 

19 



6 
7 
7 
9 
7 
_7 
43 



'I 



1Q I 1 



6 
6 
8 
6 
7 
8 
5 

. 7 
57 



,4 
5 
1 
_4 
14 



4 I _6 



5 
5 
5 
_6 
21 



3 
2 
3 
0 
0 
0 
8 

i 

3 
0 
2 
2 
2 
3 
0 
__1 
14 

2 
1 
5 

i 

9 



1 

5 



5 
4 
_4 
13 



2 
4 
_6 
1Z 



6 j * ■ 



7 
20 



100 



4 
4 
3 

.5 
16/ 

4 
_4 
8 



5 

5 

J 



1 



I V 3 



2 
1 
V 
— 

2 
2 
5 
J3 
12 



1 



.65 



-.30 



.63 . 




.48 
.43 

.56 

.36 

.50 

.46 * 



-.22 
-.0* 

-.50 

% 

r.14 

* 

-.31 



Refer to Appehdi* III-C, Test Content Description, for a description of topics included in the Core Battery Test. 
Less: less emphasis i in curriculum than test; Same: same emphasis in curriculum and test; More: more emphasis in 
curriculum than test. . °_ . ______ : — > — . . -• » 

O ctibn of Saw is total number of "Same" ratings divided by total number of all ratings. ______ 

t of Difference is algebraic sum of "Uss" ratings (negative value) and "More", ratings (positive. value) 
ed by total of "Less" aftd ff Hore" ratings. t l . y 

. • : \ .. • 57 • • • : -w-.-- — ■ 



ERIC 



The "Degree, of Difference 11 column provides an indication of the 
average 6r balance of the "more" or "less" ratings. The values in this 
column were computed by assigning algebraic signs to these responses 
(negative for "less," positive for "more") and then calculating the 
algebraic sum pf these responses/ for each test. This sum was then 
divided by the total number of "more" and "less" responses for the test. 
For 'example^ _ the "Degree of Difference" value fbr the Listening Test 



wasi 



:^}umber responding "less": 16 
Number responding "more" : 12 

Sum ' '' r?~.: 28 



lie assignment : -16 
Algebraic assignment: +12 
Algebraic sum: -4 



Degree of Difference: ^16+12 = -4 = 
A J6+12 28 < 



14 



A negative "Degree of Difference meant that the judges were mc 
likely to say that a topic was overrepresented in-its emphasis: on a Core 
Battery test compared to , Louisiana teacher education programs . In other 
words, the topic deceived less emphasis in the cutiricula than ,in tW 
t£st. A positive value indicated that, on the average, judges felt a 
topic was given less emphasis on the test than in teacher education cur- 
v r'icula with , which -they were familiar. ... * 
. • As Table 11 shows, the "Proportion of ? Same" values ranged frotn" .36: 

on the Lis telling Test to .65 on the Mathematics Tpst. . For three tests 
Q the value was less than .50. That is, the number of panelists respond- 
ing^ '*mbre" or "less" outnumbered tjiose responding "same." These tests 
were Literature/Fine Art?, Social Studies, and Listening.; Additionally,: 
the number of '"less" responses exceeded the number of more responses 
> for every v test, as the negative "Degree of ^Difference" values indicate. 



V 



*\ . ■ ^ . - ■ •' • v, *\ ' ...X...,.,- 55 

This meant that in cases in which panelists judged that the emphasis 
i given topics in the Core Battery- (differed! by inbre than S^perc^nt from 
* the emphasis given those topics in teacher education curricula, the bal- 



ance of these judgments ' was that the topics were overrepresented on the 
Core Battery tests. The values ranged from -.50 for Reading to -.06 for 
Literature/Fine Arts . 

; __ _ . .. ^ 

When the test ratings were averaged to give ratings for the. ."mbd- v 

a" - • a 

ules~~t1ie^ 

Professional Knowledge and General Knowledge. The value was .46 ifdr 
-J Communication Skills. The "Degree of Difference" ranged from -.19 for 

General Knowledge to -.32 for Professional Knowledge. 



Overall Similarity Between Core Battery Tests and Curricula 13$il 



Panelists completed the Test Content Summary Form (^pendix&iri-D) 
to evaluate the overall similarity between' ttie Core Battery tests and 

t ..... ■> - - « • 

the teacher education sequences of their institutions . Each^- judge chose * 4 

' . ; . . Z. __ J- J ':_ A': 

one 4 of the following options to .represent ^ the degree to which the test 

,» < 

he or she, .had reviewed paralleled the teacher education curriculum: 



A. The .test, content topics parallel, the teacher education 
sequence at bur institution very closely. y | , 

\. ■ ' .' •' . ^. . . 

B . \ There are some differences between the test content _ topics . 
■ and the teacher education sequence at our institution^ but 

these differences do not appear' to be appreciable. - ; 

\ ■ C. There appear to be some appreciable differences between the 

. test content topics and *the teacher education sequence at 

' ' \: ' ; '""oiir . institutibfri- -. - j . : -\ / ^ / \ \ 

^^v^^^^" ^' il^ieirief -i^^itt^* le" .similarity " .between ''the' test content topics * 
^-j-'.+c': •• ; ;4jid';' 'th$ ' teacher - education sequence at our institution. v \ 



NR v *''Nd Response 1 w^s provided . 
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■ 45 
The response f requenciejS^Are shown in Table 12; The final column in the 
■ table, "Proportion Simifip," was also used .in the 1978 Validation Study 
' to reflect the propo rtion of judges who .^indicated that the test th£y , 
reviewed closely paralleled, or did not\ differ appreciably from, the 
curriculum. , Thifc number was computed; ;^,divading the^tim '..of : :f.th^;4 ;; tod R>;V: , - J 
responses f ot each test' by the total number of respons es (A plus - B - pluS- : ~; 
C plus.fi); The- NR response;. included since it did not ; constitute 

. 1 .• , * : ;J1 . ...„,. . • / . ..' ''-r- : < ;; ~ 

The results show' that the values of "Proportion Similar" ranged - v 
from .40 in the Listening Test to .92 in the Reading Test . t That i^,; 40^1- 
percent of the Listening Test panelists felt that there Wer^e no, or no 
appreciable i differences between - the content of * th^t ^test and the 
teacher education sequence; 92 percent of those evaluating the Reading 
Test made the same judgment. For the three modules the "Proportion Simir-;V.C-' ; 
Lar" ratings were .88 for Professional Knowledge, .80 for General Knowl-. 
edge and .67 for Communication Skills . When all of the Core Battery 
tests were combined, 68 (76%) of the 89 judges who provided ratings felt 
that the content of the Cote Battery tests closely paralleled or did not 
differ appreciably fro of the State's teacher education programs. 



INTERPRETING AND EVALUATING CONTENT REVIEW RESULTS 

The panelists provided four types of data about the content valid- 
ity of the Core Battery tests. The first was concerned with J^he appro- 
priateness of individual items within the tests. The other/ three types 
of data addressed the overall content validity of each test: the omis- 



TABLE 12 



FREQUENCIES OF JUDGMENTS ABOUT . SIMILARITY BETWEEN THE GORE BATTERY TEST AND THE TEACHER. EDUCATION SEQUENCE 





w . . 

Very , .No 


Appreciable 


.(C) 
...Some 


' . ; V(D) 
* / Little. 


.No : 




Proportion' ;; " 
Similar 3 — - 




-dose— ^-D 


ittonces 


-Dif-f-erer 


lees-SSillr^ 

r ^ 1 


:-y— -Response 


^TOTAL— 




f professional Knowledge Test/Modul 


i 3 . 


■ 11 




v- - ■ . 

'* #L ... 

i a M W ta fl^f*«H « f» V * « 


l±l 1 ■ 


17 


. -- , ' ••• ^ • 
.88 


. ■ ; ? 

Mathematics Test 

Science Test ' 

Social Studies Test 

Literature/Fine Arts Test 


: 1 ' 
0 

.0 ■• 


'8 '"" 

9 

6 


: j 
2 

• 3 


0 

v.. • ■ 0 : 

f 


^•.';f\-v": : t; : .' ■: 

o ... 
i ,. 


it 

■ 10 . 
, ii 

10 


.80 \'^s- 
. .90 ,. ii 

- .67 


General Knowledge Module : ■ 




30 


8 


- !0 2 42 


• .80 • V 


Reading lest 1 
, Listening Test, ; ; - k 
; Writing Test (Objective) 


0 ■ 

1 • ■' 
o , • 


3 V 


'"'-■iA 
■ : 4 . 


0 


0 


12 




Communication Skills Module 


7 • 


...Li 


10 


i «■ ■> *f pi ■ •» w • m *» •* P * 


2 


35 ' 




Total All Tests 


12.. 


• 56 


20 


, 1 ' 




94 


.76 



Proportion Similar is sum of A'+ ; B divided by sum of A + B + C + D/.. 
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s ion of important "content topics , the .matcfi between the • emphasis given 
content topics in the tests, and in teacher education btlrriculaj and tie 
^b^r^Jl similafity between the content of the tests and .teacher ediica- 



tion programs . The 1978 validation stlidy conducted by Educational Test- 

. • ';; _ '_ v , . • _'; ' :' 41 ': * . ■_ VV . ; ;' • * - : f 

ing Service used each type of data as a criterion in evaluating the coii- ; 

* tent validity of : €he tests ; rg\riewed : at that time. A similar process was 

' applied to the tests in the Core Bat tery^ i. *^ .,. ^ 

\ "... Criteria for Special Btaview : /: ; V^.J^v ■ -ti^y-' 

- 1 - ~~ P. ...... -i" ' *. •^•'SIlSS 

. Arrange of. special revie^ v^ tlie^ 

four criteria.. Any test With observed values f allirig^within . the grange i - 
qf a special review value on any of the indicators o^ 
was tliehV" reviewed in detail; 'Table 13 identifies the ^ 

; ; . judging Whether a test qualified for special revie#. >A description ; 

the review criteria- and the^ basis for the values assigned to*'each one 
are given below. T 

* • Content appropriateness of items. An item was rated as ; - : con-^,^" : 

tent appropriate if 51 percent _qr more of the paneli^tsV V.' v 
reviewing the it 6m judged that 90 percent or; more ';;^f '^e?-' : ^-.: f ? r 
students graduating from, Louisiana teacher education ^prb^ ? >^ 

% -';'v>^. •* grams would have had the opportunity to learn the khdt^^^ : ^ * 

required to answer the item correctly. Thebreticaily ^tfii " - 

'^r^V^^! : ^.i; . . percentage of content appropriate ^itejps could range f rom: 0 
percent (none appropriate) to 100 percent (all appropriate) . 

" . - : ' Following the procedures of the 1978 study, a test qualified 

for special review if less than 90 percent of the items in 

it were judged to be. content Appropriate. . i 

• Comprehensiveness of test. The greater ttie number of topics ' -'f . : - 

cited by panelists as omitted from a test, the. less compre- 
r ' > herisive pas the test. ( , T^ls , number bouid have ranged from' 0 ' : 7 • 

(no topics omitted) to the number p| contient topics in a '■■-'[ . - 
teacher education sequence (if * the test included none of v 
these). Following the procedures of the 1978 validation 
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TABLE 13 



• Aspect of Correspondence 


Measure of Correspondence 

1 ■ «' > f '•• , ?■ .■■ "--v i : ■ 


■Theoretical 
• Range ^ 


Observed 
: Range. 


Special Review ■ f ■ ; ^ ^ 
v Values . > .VH.. 


. ■: .. [•: 


, 4 ' ,•' : -' : ' ■ ' .■ 


r..s :i: ;C. 






Content Appropriateness 
-;0f Iteis- • 

". ■ ■ 1 , ' • .v , y . , 
.... j . . ■' 

K '• ' 

■ •' ' ' * ' ' ' . 

"omprehensiveness of Tests 


rfercen^ge^Fqiief^ 
as content appropriate by total , 

lUltil JJaUcl VoVCldjjC -lUl LWv luimoj 


j 'Bttoiei 1 

f . . 


8« to tOOl' 


less than 9(115 - i. 

vi ■ , :' r :- 1c 


Number of omitted content topics 
cited by- Wor more panelists 


0 to total 

, topics in 

"curriculum 


P to 3 

V.J' '■ 


":."¥;vv.:/ ■ 
1 or more tpplcs , , • 


liiative^pasii of test? 
ind Curricula ^ 

■ ''.■.•>•* ■ ■. ■ " • 

• • •'>. /, • * •• 

- , ' •• 4 • , ' . ••• •• ••; ^ 

-V. * . . ■ \. • • • ••: ■ 
■ " ■■ ,J " 1 '" ' , 

Jverall Similarity Between 
rests anil Curricula v 


a) ProporBorbf judgments rating 
emphasis as same' • 

b) Degree of Difference in %0: 
"not same" judgments ' J~f 

•: v -(- * curriculum emphasis "less", 

' x at Ait ft* \ li'ti 1-tim • Atnn Yt4 c 4- a ^wrt I'd ^ \ 

t * curricuimii cmpu^oio mote j 

: 3 : '-r- : 


0 to 1.00 ' 

i 

-1.00 to +1.00 

■ ; • ' 'V.' 
•••• •'. >,.'••' . ; ■ 


.36 to .65 

* 

-.06 to -.50 


less than ,50 v: : 

- 

and 

■ ' ,.' ,. . "- ' j'. 

-.28 to -.50 * v 
(midpoint or below . || " 
• . observed- range) fc . ' : : f* : 

,ar; ; 


Proportion of judges: rating., _ 
.similarity as "closely - parallels" 
'or "does not differ appreciably" 


0 to 1.00 


.40 to* .92 


less than .50 ■ 



4 } 



00 
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49 



the measure chosen for this criterion was citation by- 
two or more judges . Citation by only a single judge was hot 
considered representative of the total group evaluating the 
test, .and requiring more than two citations, could have 
excluded important omitted* topics. .A test qualified for 



.special review if one or more 

omitted. 



more 



topics *^ere noted by two or 



as 



f Relative emphasis ; The first special review measure used 
4fere was the "Proportion of Satte" judgments/ (that \ the g^" 
emphasis given content topics within a test was the same as%7 
the emphasis given those topics within teacher ^ education 
curricula) . This value Gould range from 0 (emphasis differed < 
for ail; topics ) to 1. 00_ ( emphas is "the s ame iof 3II topics )" 5 , 
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JtLJshiauicLJ^ "same" only 



when they 
curriculum 



fel£ 



that the content 
were within 5 percent 
this is a conservative measure of s 
standard, value \ for What constituted 
of Same" ratings; .50 
because below this level 



acc 

was chos en as ; a 
a . test became moVe 



bfTB^~t^t^and-the- 
of one another .. Hence , 
There was no 
"Proportion 
value 

dissimilar .than. 



Similar in: content emphasis to teacher education curricula. . 

% second' measure, .was also, used 'in • • : idejit$f^^ 
were candidates f c>r special ; review, because -%^^t^^t^^es : _ijx. 
relative* emphasis . THe-^gegree of Bi^ere&ce^i^iue indi- 
cated whe.thef piiielis^ a te£t differed in empha- 
- sis from the 'curriculum by prbvidmg ie^s. emphasis (positive 
; value} than, the curriculum or more emphasis (negative 
value) . , This value could . range from 00 (all non-same 
topics^ ^verrepresellted on test) to ^1.00 (all topics under- 
represented on test). A test could qualify for" special 
review if its "Degree of Difference' 1 value was^'at or above 
the midpoint of the observed yalti^sf>£ or ; , all. tests . Since 
the observed; values ranged from: -.06 t6- ( -;S0, the special ; 
review value was set , at *.28. This figure was computed" by • 
• j adding V- i. 06 and - . SO^^d dividing the sum by 2. tests wi$fe 
(a value pf - ; 28 to];^S0 w^e "potentially eligible for sp£- 
fciaj review i This^|j»^ tests with a greater 
than average bve^i^^ topic* . Using two fac- 
tors together- -th#."jfc of Same" judgments and the 



serene e 



provided 
its were 



joint estimates 
considered el i- 



size of the 

of r,e 1 at ive eiphas is ! ^Beref or e , ; Jt e 
gible for (.special review/pnly , if they fell' within the speci- 
fied value ranges '^of -'^j^^iXt^ftAi v 



.1 



felt that 



Overall 
^ judges; 
; differ : 

. f^m^-or 

(all; j^iidied #hat theV test fclcjs&lyc paralleledi or 



measure was the / proportion 
a' : tes^^ciosely paralleled 
frpm^ i^0j0]^ijculAm» It 
juclgeS felt that the test differed 
bore; little similarity to, the . curriculum*) to 



of 

or did not 
could range 



1.00 
: not 



. " : . • ' : ' ' ' : ... 50 ' 

• differ 'appreciably from, . the curriculum) . : . Tests were :. ' 
f selected, for special review if. this value was less than .50; . 

that isy if e f ewer than half of the panelists judged that the . - • 
. overall^ similarity between the test and the curricula was , 

high. Th e, same criterion' value had been employed in the 



-I97S"- study. 



Special flevteV^ Criteria Results 



The observed lvalue for each of these criteria was compared with 
the special review value for the eight tests cbmpr is ing 41 the Core Bat- 
tery. Table 14 presents the results . Four tests . ^Professional Knbwlr 



edge, Science, Social Studies, and Reading) had values within the 
accepted ranges on all criteria. Noiie of these tests- warranted special 
• -review. ;/ ■ . ' " . : 

The Literature/Fine Arts Test had an: item appropriateness value of 

• •; . " " '•. • ' ■ • • ••••• ' •• -V" " /V •- ;* ■ . ; •• •. 

84.0 percent. It was,, the only test requiring .fuxther review- for item 
appropriateness . Although the number of items was , not large enough to 
' support statistical testing bf distribution, five of the 10 items: judged 
inappropriate concerned a single tbpic. This w^s tfievtopic of "relating 
works of literature and fine arts to one anpther ^ 
torica.1 

fftShematiSs-^ and the objective 

' Subtest of the Writing Test qualified foxv special review on the basis^pfr 
comprehensiveness , since each test omitted one or Idrej topics cited by 

two or more judges. None of the tests fell within the range of review 

. •• >:•••..' . • . - :. ' ~ v , •; V /•', . :. ;• * • ' , • • *i . . ' 

values : for both factors in the relative emphasis criterion. The Listen-; 
iiig Test was ^ the " oiily one of the eight tests to -qualify ^f or special 
review on th6 criterion of overall similarity. . , • ! 
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TARLR U • 



Vi, ■■ . • 

' 'lip; •(;-.':■'■. 



>1- 



SPECIAL REVIEW CRITERIA DATA F0R;; CORE: BATTERY TESTS 



.tr • 

* 


Appropriateness 
(< 902) 


topic .' 
omitted) i '• 

\V f. 


• ... ; — .. — 

..i : _: ... 
Relative Emphasis ' 
(< .50) ., (-.28 to -.50) 


Overall Slmilaritv 

km 

■ — ••' '■ '•■;^>'r, 1 ; r 


_ ^ ',>_'' . ' '. 

Professional Knowledge j 


92,9 ,., 


i w: : :■■'/■ 

: • f 1 ' 


.5? ' -.32'* 




Mathematics 






M -.30* 


■ .80 ^5) 


■•• ■ k ■ : ■'' ■ 
Science • 


96.7 


0 . 


.63 , -.15 


.90 


' ' .•''(■■ 
- . - - - - '- ' ■ > " " ■ 

Social Studies • •'<: ' /• 


100.0 


0 


.48* -.-22 


<y .82 


riterature/Fine Arts r\ 


82.9* 


1* 


.43* -.06 


" . ' ,67 ; • • ; 

/. 


Reading >''.'■■>'■■•■' 


' 100.0 


.0 


, .56^- ' • i -.50* 

' ? ■ \, 




•" V; '. " « ' 

Listening , •. - • ' 


- V : ' . 


;.: 2* . ' 


.36* -a*;. 




©InlfObiective). 


_ 95.6 


... i* ; 







A lest fell within range for special review on this criterion 



Application of Special Review Criteria Results .>V 

The impact,' ot the review criteria lay in each test's contribution 
to the cbntentf"/vaiiiciity . of il the Core Battery module of Which it was a 



component. ^ Thai's was-'" because ^scores are reported for entire modules 

-i ^fe^^:.^, * . ' ; ;. . ' ' 

rather thai^^naividual tests or the Core Battery as a whole. Therefore, 
in - evalul^j^^^le : ' impact of . the "special review criteria, it was neces- 
sary to ; ,a^ign; a .niomerical weight to each test^that reflected the mimbejc- 
of cx^^i^^^^U^^^' the test "qualifigdL^fdr special reviei^r and the 



tip . the total module , ' |^jis ' IW^'^^^^ep- porocess , 



Piled' in the 1978 Validij 




in t expi e t atlons wfere: ^ 

0: .No criteria ifor special 'rwiew, test very closely related to 



•;. cur^xqMa.' • •; 
' ' T: . Oi^ '.'^plt/fer^ion for specihl "review, test ae losely related to 
> v > .■■" ,H - .. :'vYc^ri?i§eila. • • "■' ■■ v 

2 : Twa criteria for special review, test reasonably related to 
: ;■ : curricula. ' •• • \ • .• . • ■ ■ - ..." 

•^3;;^ Tforep^piteria for special review, test probably related to 

r : OairiCUla. ' :;: y, . ... , : ..' 

v X\ l£ur; criteria for special review, test not related .to curric- 
. uia; ;'/^' ; > ' 

« The sefqpnd step was to Multiply the.yalu6; v given each test by the number 

'. ' I , i^/ ' . :■: _ * '#'.'1 ' . _ _ __ ''. " ' * * _^ ". 

of items iii "that ' test and to sum the resulting figures for all tests 
• "within a module. The results are shown in the "items x Value 11 column of 



Table' 15 . This sum was then divided by the -total' number of items in a 

. '. " • ' ' ; . - • • ... . : " • ' ■ , : - V '»^^ ^ : 

.^odule to produce the Module Value weights^ shown in -the Table^l^Iri this 
way , the content' validity results for each ."^st contributed as much* in 
judging .the^ content validity of the modulef|^ performance on the test 
would contribute to the module score; > ; ^ tV " 



. TABLE 15 



CORE BATTERY MODULES AND PROGRAM CONTENT 



■j I 

' r ' ■ : 
i ; : •' ' '»• . 

/,.... 



: of Items. . 
(Form A and ' 
: Form B) ' 



itm 



Value 



Interpretation 



professional Knowledge 

pi" ' " ' " 



, Mathematics 



'Social.: Studies 
Literature/Fine 'Arts. 
' - TOTAL 1 

Communication Skills, , 



.Listening •' . 
Writing (Objective) 
. TOTAL ■ , 



103 



25 ' 

30 

30 

34.5 
119.5 



30 
40 
45 
,115 



,1 
0 
0 
2 



' 25 
0 

! ' /0 

; • .69 

; I 



. ; .0: 
80 
45 

: 125, 



Very Closely Related 



0.787 



1.087 -i-Z 




-rr 



■ - /■ ■- - . -*- r ■ " " ( 

Module. Value equals. Items x Value divided 'by .Number of Items . ' 
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.0' : 



:. -1 



■ -V Both the Listening and Literature/Fine Arts Tests had a .w.eight of 
.2 (reasonably related). For Mathematics ; arid Writing (objective subtest) 
-.the weight was 1 . (closely related) ; ^The other four tests all had 
weights of Oi-^r-v^ * \ 

When these weights account for the cptfttibution 

: . of -each test to the total score of its respective module, the,"^dules 

• ■■ demonstrated acceptable levels of cbriteri£ validity 

edge was the drily test within its module, and it produced a/] module 
weight of 0.0 or "very closely related" to the . teacher education 
sequence n iii I^lsi^a. * Tlie other two Core Battery modules , General 
^^^•vKfibw ledge and ^Cbmmxuiication Skills, had weights of about arid 1.1, 

respect ive ly . This was close to the value of 1.0 .'Ehat^wtfs interpreted 

^ ; V ' -' • : • ' . .• . : ■ . ' " . .<> : • ■ ■'. ' 

S.V'V -*:^'V^ V '#s&" c-losfe ly -related" to the State's teacher education curricula, 

rr ' v'^y^^^^phese interpretations are riot meant to discount content -differ- 

J \;; - / . / . toc^:: between Louisiana s teacher preparation programs and the iridivi'd- 

s^8^4? ? ^B?^^^^ . __ _ _ . ' ' • . ... 

• - ^ Battery tests . They simply provide another way o<f evaluating 

. the* content validity of the tests ' arid, by extension, the , total modules; 



It should be stressed that the methbdology arid values used in the spe- 
cfal review criteria are riot absolutes, but were adapted from ?the. prec- 
edent of earlier validation studies . ^The -special review iiiforn^tiori; 
therefore,." is presented «to the B lue Ribbon Score /€ommittee and to the 

.- ' - : -^.^'V: :. ; /;-' i ■ tft; .. • ■■ ■ : ■ : r: 

' Supe r int endent of Educ^ti^nwor : their use in making informed judgments 
^about f the conterit vali^^j^of ;; the Core -Batterv.v * *' 



^.-tf 



table is 



NUMBER AND PERCENT OF QUESTIONS JUDGED CONTENT APPROPRIATE, ' 
; : . BY TEST FORM AND HALF-PANEL : '* 



j ,V.- - 

" 




loss-B 



Number Percent Number' Percent Percent 
Of Items Of Items Of Items' Of Items Of Items „■ 



98/ 94.2 



97 



95.1 94.7 



Panel 2 



Form A 



Form B 



it 



mi 



81.4 80.1... • 



, 23 92.0 25 .•: 100.0 96.0 



.24 96,0, 23:- f: 92.0 94.0 ' 



Science 



30 100.0 



30 100.0 100.0 



28 93.3- 28 93.3 93.3 , ; • 



30 ' 100 ;o 



30 100.0 100.0 



26 86.7 



29 



96.7 91.7 



Literature/Fine Arts 



29 82.9. 27 '1 79.4 81.1 



28 



80.0 24 70.6 75.3 W. 



30 100,0 100.0 



30 100.0 28 , ,i3.3.' 96.7 

'„, . ../Mite . : — ■ 



33 



82.5. 37 92.5 v 87.5 



m.'- 97.5 ; 97;5 



Wrltinp, (Objective) 



41 91.1 42 ' 93.3 92.2 



38 



84.4 m¥ : Mii 86.7 



*'• 



>?4 



73 
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• •acr©s*s ■ t^S^ or f $wei f&ihts. V--^ese /¥esulti' r are' ; 

similar to those reported in the 1978 validation study; V : ^ . 

table i7 provides Information about the extent to which thecal f- 
• panels agreed. on the classification of individual items. The columns in > 
the Table ^report the percentage of items that both half -panels %JLt ££*e ■■j-;!- 
appropriate, that the half-panels disagreed lipon, and that both half- 
• 1 panels felt were inapp^priate*:^: Finally,* • the columns ide^ti^ ied as • 

"Percentage ^ of AgreOTient" contain the percentage of items that both - . 
half-panels agreed upon as *eittej->j^^r ~ f 

Tlie percentages of agreSmSnl^ ranged ffomV74,3 (Literature/tine :V 
y|^^;u^'^ts) to 100.0 (Reading) on Test Form A. The range was smaller for Form 

B: front 82L& (Writi%)^to; 96.7 (Social Studies). Of the 16 percentages 

*W^- •- ' - — ■> v 

\ -Nib of agreement calculated, i4 were greater than: #4 percent . By contrast;,- - 

. '•' • : . ; ..; . : v • ' /: ' • • . ■ ' . ' " . " ; * _\ r --x&g£ 

oily 29 of the 46* percentages of agreement calculated for the Common and 

v ... ■:•.•/. : : \ • .. * ? 

, * Area Examinations in the 1978 validation study Exceeded 84 percent. ' 

An average probability of - ; agreement*- was also calculated. This* • 
probability was estimated as the average' of two .conditional probabili- 
ties that are^ given by: a) the percentage of . items judged content appro- ',' 
priate^by>both Panels 1 and 2 divided by the percentage of items judged . \ 
appropriate by Pjiiel 2 3 and b) the percent age of items judged appropri- 
ate by ^ P^els ^ and 2 ' 9 divided by the pe^^^age of items judged appro- 
priate iy P^el This, statistic is a measure of the probability that' 
both of th'^^^^-ptoej.s would judge an it em- appropriate. — ^an$le 
•below shows how the average probabilities of agreement were calculated 

J: - ■ ; " ? *" ■**''. 

■ ■ r - - ' $j£-- ' • ' _ _ ■ __: :_ . 

for the Listening Test,: avenged across' test forms. : * 



O - ;/-- : „: : fe . V-l: :,^:- v. ■ '. - ; 



eric; 



• .11 



■ v - 



■i. 



o f H 



TABLE 17 



Wr 



AGREEMENT- BETWEEN HALF-PANELS ABOUT . ' : : 
CONTENT APipApi| OF ITlSi.BY TEST. FORM \ 



./f .- ' -'-.V 

•.x Sj- ' <i''.V 



j' '''''' 



Both , 



Appropriate 



Form. A 

[ . Both Percentage 
Panels of . : . 
rei Inappropriate Agreement 




Writing (Objective) 



Form B 



Both 



I 



Both, 



Percentage * 

" of ., 

Appropriate Disagree Inappropriate ^ Agreement 



•: 84.r ' 17 .8 



; 3 , r- 



, - . in , .1,. ♦ ' 



>"'}, • l!> .\ ' 



■^■v '''' ; ' 



■i ; 



4 t . 



.4 • . J . 



; • la- . 

• ERIC ?■;' 





V 








. T,u llliVl .. 




■V ; 












■•'\. . i 






"'' • .. >-,■■■ 
















'? ' ■ 


































4 i ; . 


.» 






•':.v ' 







ju.' • • v frs V:., 



■•.v..;" a 



1 . 



,73: 



V.:- : V;. ■' '. W'.iY- 

; ■-,L^,:/ .... . . • .., , : ^ --_ ; 59^ 



Percent-bf Items Judged by - 



9 . Panel 2: V v Appropriate Inippfppriate; t 



_ M 



Appropriate- 86.25 ^ • :ilt?.§v 

Iha^propria^e ' ■ j\ 25 . ■ — / > ; :U '25 " V- 



' ; . Total , > * V ,87 .50 . 12*50 



597.50 
hi. 50 



IIOO.OO 



- Frq|i .tfiiS 1 ^ table-, the conditional probability of agreement for.'. 

.;• items judged content^appropriate was .commuted as:' .■> ■'•.■>.•. .: '< ., ••; ' 

' ' A "V* ' ' .8625 ' .-X.8625 .•../; ■ '*"Y\^ : -/>(' '.V.. * ! 

>;i-.i,^.-.^:;;.: . • .. ' r T975Q ', + ^8750 ; _ Q _>. T-\ - ,. 



In a similar fashibqs half -p^anel cohsistenpy^ in agreement about 
items judged inappropriate, was viewed as ^h^k probability that ufie of the 
panels would judge an item inappropriate; if the othpr panel had judged 

* * ' • P ■■ •.■"•■» i ' ■* : • 
' L _ ■' ; I. ■ _ ■ _ ' _• ■ ! ; ' 1 " ' * ■ ' 

■„ ' _ • _ ; ■ t ' ■ _ ' t , ■■ ■ 

it so. For^Jfhe.' Listening Test, this probability was: , 

^ ..... ■'• ..0125 .Q125 • 

/ '* : ; . .0250 .1250 « " u 

'.V'*- . : : t\ . " ■'■f.:;' -, .300^< .. V - 

% -The;, conditional probabilities of^ agreement n for all ^ of the Gore 
• ; '>:Batt^ : ^'^*^^-'are. presented in Table 18. The ^value's ' £m| the • items bias- 
siiied content appropriate were consistently high, ragging from .87 • p 
.98 . In other words, if 7 the judgete on one ^ .half -panel determined that -an 

•. • ■/ . :\ - ' :• >•« h 'V; : 'VV j'l' - '-.- 1 ' ' 

item .was * appropriate/^ the judges on the other half -panel generally^. 

agreed. " • ■■=■■;. "T-,/-.' ♦ . » '.. 

-- ---- :-— : . • . i- ----- . 

On the other hand, there was less -agreement about items that were 

# ;g • ; • - . • . ---- • - . - i — , — *-t- '^>S. ; 
judged inappropriate. The values ranged . fttm .00 "for three" tests (Math- ^ 

* ematics, Science ^ and.. Social Studies) to .63 (Professional Knowledge.) . , 

Only; three of the' eight tests had ^ah^ average probability o£ agreement oh '; 




•f..^- vv. w: 



6« 



I-, 



TABLE 18 



'AVERAGE PROBABILITY OF AGREEMENT 
"ON ITEM APPROPRIATENESS ACROSS 
' HALF-PANELS a 



■ - ProfessionalyKnbwledge 



Mathematics 



.92 



Science 



35^ 



.97 



Social Studies 



.96 



Literature/Fine Arts. 
v •Readiftg ^ 



.8X 



.98 



r 



Listening 



.94 



Writing (Objective) 



.91 



Inappropriate 



.63 * *. 



.00 



.00 



.50 



► 30: 



£3- 



Note that the data used to compute these figures are based on the 
average of content appropriate^ and inappropriate percentage responses 
Forms A and B.' Refer to .text for an example of the calculations. 1 



I ■ a 



for 
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61 



inappropriateness that was \50 or greater.' These lbt^ values, point ; %o 
i? *' bne^of.lthe .difficulties that arise / in constructing a tfest to measure 
- Q'X- knowledge' of the content among different teacher preparatory programs . ^ 



it. items- will match the' Coritertt of most curricula; ^others ; will hot ; 



bjjtr nbt* all test 'itl&is will -match" all the topics of all of the r curric- „ 



'-./''■^ It "is interesting to note that the tests on which there was* .00 . 

V< .,■ ';'[„ yy'^-'n' • - . • • ' • __^1".'V- - 

agreement- with regard to content inappropriate items (^atbedatics, Sci- 

ence, apid Socia J. .Studies) had very high probabilities of agreement- on 

< 'V- : : . " # - . .-■ .J:;.- . * ._ 

appropriate "ratings X -95,- .97, and .96' respectively). • The panels for 4 " 

thesis tests were composed of members from the most diverse subject^ 

' - J ' ■ " : . • - ' " " , r >" . - ; . .-" ; " ' ' " % 

areas,, who" apparently agreed on appropriate items , but disagreed greatly 
- tin \ inapprop r£at e ' items . The disagreement on inappropriate items o Very 

likely re|lects the diversity in curricula among the pane lasts 1 univer- ' f ? 

sities and departments v" J 

High average o pi:6paBility of agreement for inappropriate items 
' . occurred for the Pr of ibnal 'Know 1 edge , "Literature/Fine Arts and Read- - 

■ ■ « .--•::'•"».«•;>. 9 H ./■' " V, '.. „ ; . 

-' - " - <» v • ^ v» ' " , i ;V :.'----r , >- r ".-.«'"- ' y. ' •- ■ ' ■ • '.. ' " - - ' 

1 _ _l • ' "_ 1 ■_ * _ / ■■ "" '*'*•'_■ 

ing Tests ( .^63/ iS5| Vknd ,:50 respectively) . This suggests that there 
.was' closer agreement Sbout.^the content of these fields across the uni- > 
versiti^s and .the departments* represented by the half -panel membets for . 
tljese tests. "than among other panelists. 

> - ' 



^9 reported .ear liter ^ in Ta^le ^,10, the analysis of the relative 




r emphasis (Df tests' aiid^ curricula, focused on two Values : (a) the Pjopor- 

. : " ' ■> • ' ' ' . ; ' " • ' - «r - . . • 



tion. of. Satfie," whWE: indicateid for each test as a whole the percentage 
of jj'ansiists who;.: .fi^^^l^e ^tppiics , received the same ^emphasis in the 



teacher education^Qurryc t uluiD as : in the test ; arid (b) the Be gfee of Dif- 
ference," which indicated the -average balance, of the "more" or "Less" 



- - ^4 



ERIC 



ratings. The half-panel/ analyses shown on Table 19" compare the half- 
pariel ratings on tliese same two criteria;. 1 vtiP' 

The "Proportion of Same" ratings show a high agreement between the 
Jiaif-pme^ F^roportion^^ 7' percent or less) for four 

■* "'■ ; *'* - - • _ ^ '"" • " • 'r ' " ■.'_„„_ . ... 

tests" Mathematics, Science * Social Studies, arid Listening. .The half- J 

■:?,> / . - ' ■ ■ > .' ■ ■" a ■ ■ " ' . - - • 

panels;. 'for these examinations agreed on the proportion of topics that : . 

• 1 : __ _____ __ ■: ^ 

received < the same* emphasis on the test as in their teScher education 

jcurriculaV. % There was somewhat less agreement betweeri\.the half -panels on 

* -V. :- fJ : , «Q % : . . . ' \ :: ;r - .-' , ' ■ " 

the - "Proportion of Same" ratings for the other four examinations , rang- . . 

ing from a difference of ii , percent for Reading to a difference of 28 

percent for Professional Knowledge. 1 * 

-The resultsjfor "Degree of. Diff erence" ^idicate Jthat the.^half -pan- 

els agreed in 4 i r e c t ion'a 1 i t y ,fi^e of the^ eight tests . These were r 

the Reading , Writing, Eiteraturfe/Ftifte Arts , Mathematics,, and ' Prof es . - 

sioilal ' Kriowtedge Tests, * in which all half -p^ner^bjngs vfere negative " ' 




except^ for a .00 rating produced f rojjp-one of the half-panels *in Litera- " • - 

\ ■ - - r : - : _ • 

ture/Fin^Arts. Both h^lf -panels agr'eedi, on* the ^average, that differ- 

_• .•';■•<;•; • ' . v" ■ ... ; • ■ ' . - ^ . i- * ■ •" , ; . . v ^ •. 

ences in emphasis Reflected an oyerrepresentation* qtf content top%cs on * 

these tests'. ■ ■ ..... , - - ££* : '- - -"*HL- -^f^ 

.""."i There were* disagreements in "De^r€^e oj. Differ encfe"^b e t wej^ri f ^ 

half -pane Is . on the Listening, S.Qie iyago and Social-^Studies Tes^.^>Jt^ is j."; 

interesting that these all _fad high half ^anfel ^reementr on' the "Proper- 



- ..... 



tr ■ 



3 TABLE 19 



HA£F r PANE£ RATINGS . FOR RELATIVE * EMPHASIS - * 
V OF. fESTS AND CURRICULA- ' * 



Panel 1 



Proportion Dj^gree^b.f 
of Same — [ - — Difference 



Professional Knowledge, 



Mathematics ' 



Science 




Social. Studies! 



:.v#3£v.5<* 



.4b 



Literatttfye/Firie Arts 




-.14 



Reading 



-.71 i 



Listening 



.33 



Writing (Objective). 



Panel ,2^, 



Proportion, ljjjegf ee of ' 
of Sage 



.42: • -,43 , \ 



-.20 



.62 



-.41 



.46 



^.69 



,33 V'.; .00 



."50 



*- . 33 



: * .58 




ahaljr 



e data were generated in . the -Fall validation procedure by .*perso 
reviewed only 8 Fb^m A of the LlsteWhg^est 1 . The half -panels f dr tfiis 
Vsiiigkere* those* cons tnicted^oJEj Form A^of the fc 



for an gffitflana^ion o f as lljzniaen t tJS haEflroanels for fefce 



test. See Chapter II 



^flan^ 



. .' of* 
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tion of ' Sitae 11 . fat^^^^' It Spears . tfiaib-^he Mlf-panels* for these three v 

^^topics that received the same" 



v,. t%sts agreed highly, on the 



Emphasis- in their ,.< 




of whgther those topics that. * 

reoeive'd ir more" or "'^ftSsS. 11 fcmpha&is * ±rt thei^r ;c3^:icula> - : 

"'- .^..-:v':'; '. '• - ? : »-J V ' S * /■'■VV , •'• ' • • . * 

• >pS the other h£nd, ' half -panels with relatively higher disagreement 

•__ . : ' - • ••• • *■ : f • ■ - • *V- '• *-- 

on theV- Proportion of Same ratings (Reading,. .^Writing, ' Lite.tature/F 



me 



Srtss aid, Professional Knowledge. Testes}" Agreed 6ii the directionality of # .< 
^dif f erence for ^^l^se topics th^::^r^$ ' ^ . 



: -It app^ags- thatr. hi^\ agreement l^^W^en 



not »bh* ho 



,jxi either 

..... y>£ .'■ _ :;. - . A-y' ' » '* : : 'v^*' C: 

;bf S^e^^atings^b^ 



• half -panels/ for the Hathem^ticB^.Test 



ones/ 16: agree ^hlghjy ; on" :: ^gth ""Propprtion' of Same" and -'Degr^^f Dxff «r ^ 



. encev 



SUMMARY 



* The results presented- in this > chagtet, ^suggest that *the 



nt- 



Review Panels generally evaluated the Core Battery tests as appropriate J 
measures of knowledge for teacher^ education graduates in Louis iana^/ e 

- • ' " .. ; . .. m ^ • . • .• ; fy ' ' * ■ 1 

all te^ts but Biterature/Fine ^Ajrt^ the panelists rated^SO percent ^or 



'more d£ the^it^ms as appropriate./ It Should "be^emem^erid 



te by half or Hgore of the 







aot be inrfP&deit in cal^culat jng^ the A Knowledge Estima^ion^^anel 1 s 
r recommended ^ performanQj^^e^el for the jCbre Jatt ery j lit £oui" pf- tfie * 
eight^tests two or- more jiidfe^, dite^atbpic as omitted.' Hpw^Ver^ in 




7T- :r 




$5 



•J. 





se tests the number of omitted topics was still s: 



ranging *frbm 



, The "Proportion of /Same-" ratings ''snow 'that for ; the majority of the tests 



of^ eigljt') the panelists also judged that the emphasis- placed on . - 
;^ dbnfent, topics in Itheir teacher education curricula was acceptably ©IpS.e'. : . 



* • r to the emphasis given these topi^ in the Core. Bat£ery.v VV 
The numerical weighting /used to convert _^ie panelists 




.eivt Si': i* 



on a variety of content validity measures to a single xatihg :^>f !-< 



^ ■ module's overall appropriateness (see Tablfc 15 ^-demonstrates* w thW^adcept-.; 



" . abirity of the Cd^e Battery. -The' Professional Knowledge Module was rated 
■ as "Very^ClbseJy Related 11 to the Louisiana^ teacher education . cur ^icuia^ k - 
v fioth the General Kifewledge ' and Conuffunicatipn ;*Skiiis Bo^ules Ve^i: r evaiu- . 



^ V-; ated as "Closely Related. 




CHAPTER IV. 



KNOWLEiSgE ESTIMAf Ipfe.RESULfS 




^INTRjiblJCTlbN * 



fF.: -;-. . • _ .... .• v ;v ■ 

The purpose of this chapter _is^ to present *be v analyses of |:Se judg£-l 
i -j mentft tfre Knowledge * E^imation P ane 1 that ^rie 1-ded • in^^rnat^on* . fbr deter- 
mining the^ performance ^taR^rd on the J NTE . ^ore Battery. The .chapter is . 
• * divided intc? f bt$r major s e^tibns. The. first sectibi contains a discussion 

* •'•^ebn^ 



3£. 



of standard setting an8-^&t%4^^ The ^^jcond Sect^t|5&): 



^ains an. overview of t#e khowledg^S^timation process use 



^ Included are an^explah'a^bn of ,the purpos 



ST 



; f;^eis , a* review *of j the training provided to 

and an explanation of the -relat, 




rds • that ; Resulted fro 



t ion off thejl^Jhapter corit 



/items ; -panel ^result 





Knowledge 'jEs.tlmaltion I s 



"V 



descripraonlof the; t'ask£ 



among the vaxfious stan- 



ledge/ Estimation ratings. 



^tKird *$®c£ 



yia., in 



<r IN 



final secticJ^'of the 



ts of ^he /panel, jarkttogs, bfe the test 
Estfematsion Panels ^ar:e^^eporte^| 

■ I- 




CTION TO, STANDARD 





% The tise' of tests' to* classify examiife^s i^b jbat'fegories or stat*es^>ased 
/oh. test perfbraahce has incf^Ased ^in^^the^ last v 15^ years. Tests 
used to make" decisions /ranging frpmvthe fonnatdon 
.. certif idS^on aifd i^Masure ^J? ind&iduals desirjS^eStrancfc^S^b • vai 




ess ions. The ^e^pf ^t^j tftt . f feur certification is an^Ami 



o - ^ 
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ter use, of 'tests; One feature that must be preseftt when tests are used t© 



' make clas^if icatxon decisions about examinees is a score that represents the 



decision ;^ point (the point that is useaStb separate acceptable from uiiaccep- 
table candidates) ; * *The issues ^ ^ow to establish qualifying scores aitdjj^e 



validity of the reciting classifications have been 



debate in the psychometric^ community^ 



serous 'methods have been ^prppoSed for settij 
tions^ of many of the methods llkve bieK compiled by leveral researchers (e.g. 




Hambletoa & Eigndr, 1979; Jafeger , ■ 1976; Popham, 1978; Shepard, 3,976). * fine 

• ■ '*< » _ /-ivy . \-\ . . ". \ . / \- - ' ' ^ * ^ . j- 

common ..feature of all-^standarct setting gffl et hogs^is Shat .they ; : employ human 



judgment. ,A11 s t and^rdv set t irig' ; methods , - . ev'eqf the \ ^fl&S #/^ at use .test data 



t , involve 



; ^jfiust, at some possigt^ 
^ ' the standard.. ■ This ' ±& not ? 



subje^tijle ''judgments** about} the' . final placement 



^fcnec^&r f ily undes irjfc 1 e 'condition . ; * 



^ . Miny ' u^ in* areas other than meifetfrement are sub- 

jective. Speed limits", accej^ablje discharge limits, of chemicals from indus- 



pTantsy a|id numerous i other j 



,| w - . 



:' 7 ^esM^'; (. subj^otiveVJ^idgment 

. Sine v§WBBards" are ^gener^l ly ^ihav^i 



rd$ that, affelc^ daily life, are the 
a 1 IVsob i eet ive/ cx i ^ eri a. ^tp . 



i : individuals* whos e r j u4gment 'as^ solr 

metits re^ire^f bf them 
jf; jing scores on^h^ NTE 

f essors who' ywere ^nominate 





are 



to meu 

ase with . the dacerrfination of* qualify- 




v The test litems • 



revxewec 



pro? 



l tlje ra^ifigs 



perform the ratings . /Likew3 



t^i/.respeGlsiV0 ^dej?mis as being ^qualified to # 



Blue ^-Riqbdft" S^bre Comffiit- 



"fee we?e feiectfed f6r th^Gbhtr:y^ib^ 

^ence g^jild "Bring to ^he task.\^^^ *. ' J ! / 






Hambleton and Eigndr (1980) have classif i"e ? d a, number -of standard get --^ 
ting methods .into t^ree categories : Jiidgmerital, Empirical^- 



Empirical and /Combination nfethods requite actual tfest.^data ajid, iri most .' 



cases, some external measure 



examinee's status to determine the 

' * ' '\ ' ' ' . * ■ '• 

placement of - the _^taii$ard. For example, the Contrasting Sroups method' (a 



Comblnat ion^tfethocf) ■< requires that D ^ judges (usually instructors) rate each 



- /e^itaineeras abov^of b£loW the minimally dcceptabl«e,.performance leyfel.. Test 
\ scj>rV. distributions of the two groups* can be c65f>a?eji* The point of inter- ■ 
ion of • the two distributions is *a, t logical choice f a pex|prmance , s-taja- 



1 at ion . who would 



fore, one ,ofl thef. 



esS methods require - 
scbx&V - Tke three, igpst 5 * ' 
^ky . fe?54) , f Ahgof f 4 * 



Nedelsky : 



"the 





d^rd%- : 

- "■' • - -{^ ■ - -\ ' 

*" Sueh ;data werer no* ^available for 

the NTE Sore ^Batter^ fq^ teacjier^c 

Judgmental mettipcts jo£ ;s^t^^v^t^yaard^>. 

.to rate test 



cominphly menti 
<1971), and Ebel ; <i¥ 



Each methoicFi.^ obrief ly < described below! 



The Nedelsky method requires* 



to 



e m^imall^knowleSlge'able examii^^ jof 
test .item. 

<ih ^udgJ then eliminates .the option^ - he ol%slie 
felrlyCiiicorrect by 



i -\ • i\ \ .... 

actor of a multiple choice^ 



ERIC 




f eelSi are 




69 



'i of th^ minimally knowledgeable persoix s correctly 

& ■ 

tsV'tb^, reciprocal of- the number of options remaining 



S^^S^teteenfi^ififrinated-; Tor .example, for, ;a S 



feel' that the minimally knowledgeable student would 



elimiij^St^mfoptibns as clearly incorrect ; •: SiSpce -tfcjae. options remain, the. 



'estimat^ answer is one third (1/3) . 

: One feature of ; the" Nedelsky method that caii iie> problematic . i§ V.that 

•> judges -.V* ratings, result in fixed and unequally space^ probabilities • If a£!W 
incqrredt opt ions ^afee eliminated by a judge , ' the probability of a correct 



response is 1.^^*5 If two options remain, the probability is 0.50. :-i 'Under -no 

y cirqumstaiices may the prolDaki^ity of a correct response be^^S *OY laky, btHer^ 

. '....A;-, r. .... _'J : 

number between 0.50 and »1.0Q; This restriction in the Method has been 

|- ^ regardedCas a drawback. H^wever^" the Nedelsky method does require, judges to 

v r, rate ; each dis tractor- Judges must consider th^ fine distinctions : bBtween 



distractors.- that may make them differentially attractive 

J , • ■ -* : %&%t . ■ . [y ^ 

ard set tin; 



xaminees . This 





The Angoff method alstf requi^s judges to consider test items from th* 
f r3tae * of reference of the 'minimally 1 knowledgeab le exkmifee^^ ;• In the Angoff 
method ^ • however^ the judge i\ required only' to estimate tha p3fbbabil£fe^/ that 
the minimally knowledgeable s tuden1\ would answer the item^_correetiy In . 
effect, this method requirjl^ judges to predict the item difficulty ^.ojE.^eiacfr 




1 ' .4^ 

V..-- ■;. 



it^a| for the minimally knowledgeable group^^ "examinees . The* stun 



leTse 



probabilities equals the prdcii&feed ^est-^score for .the minimally Jtnowlegeable 
examinee^ 
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The majbr^criticism pfthe Aiigbff method is the fudges 1 inability to • 
item- scores for examinees (Livingston* % Zieky, 1982). 

. - . - ^ -• . .Jp : ■ _ -'- -- 

This is an unfamiliar task In most judges, and studies to- date , have not 
. indicated that judges can do it accurately. While judges. 

test scores and* can often predict the overall test performance of an exanfi- Jt 
;nee, it is much moire difficult to predict performance on a ' specific item. 

: A modification of the Angoff method/ W4s; used in the 1978 fflTE; valida- 
tion study. £n Louisiana and in other studies. Judges, "were provided ^witft § 
' sfven-p'oint scale (*2, 10, 25, 40, 60, 80, and 98) and were instructed to 
choose one of t^e points to represent the predicted probability of a correc^ 
4 answer - Also inclu^d in the scale waS an. J f I Don't Know" option. - Tfiis 
method still requires judges 'f%& predict item difficulties, but the scale 
limits the options available • and restricts the predicting of item diffi- 
culty'. Additionally*" ' som^ %esearch^:s maintain that t he, premutation of a > 
v . fixed* sdaie '-'may ijfeiue^e , judges to njakq, certain, tjrpes of: ratings. Fjcgc 
ri)b^exampl^^on ^^^tionnaires^there is often g, pendency aifion^g respondents to 
choose bptfbns tgward th^ /middle of the scale £3 Similar -results may occur 



3> 



v The Ebel (1972) ^mejl&d s is a nip re 




method than 

a ^ 



|off . methSds »^ St ^nv^|e^ mating ^ach; tes-j: item on tyo dimension! 



♦vance and- di|f Icurg^ Eb^X s _ original" pr. 




'tion O/^fth'e mJlnod \ised four 



: * . ., i .-^r§lfcvance'^fe ! StSga!n.es 
V 'and thrifefef difficul 
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of categories Within a dimension Is riot prescribed. .-■ Ohqe .eacH item has been 
j, rated along each of the two difflensibns^ judges . assigh a percentage to each' 
cell fbfil^^ -and a'.dif fici^^f^ategofy . The^g 



percentage is the judge's epcpecj^^^fe^f the p^lol^&c.e' of the minima lly,^K' 



: ,Iqiowledgeable^.examiiiee|bn k;rtflf; ^ ^ " 

^The EW&Wiriet^^ use* in this sti^ for several rea- 



combination < > 



son^^^^^^t, the Department of-^^ucation's researchers felt that 'limiting, 
the^^^^^^f the study ratings of item- 
(easy]^^aerite, and hard) would^yiel 




culty ta three broad categories 
Reliable and - valid data than 



^ ^wotild . the Angof f : s evfe- point ^^a ting scale.*. Second^ there were no 
restrictions on thp; assign^ as in the N&Hsky 

and modified Angt^ff methods. • Ju^ges^X^ere free ^to choose any percentage from 
'0 tQ IjJpVfor each^fcategory. Third ,*^h^ task of estimating the 'proportion of 
.f'-.corrjpct scores Jy group of similar items isviiot unlike estimating 1 a v >total^ 

it percent sco^v Since this * is a i eve l-> o f • eva luat ion more familiar tor fc'f 
\j^4^es 9 it^was felt* tfrat N 
\ v l^k^^^ w^ltl tliat of . rating^ xndiviciual it&as..' Finally, .th^ p^^^i S^f 
.xel^vance dimension tblthe 'difficulty dimension (the Qjily ^iqehsidh iated:;** 





0 the IffSB \ Edudational" Testings Service study) appearecir 'to ? be . a sigr ificant 

e^ha&G6ip^^ . of, tfi^ previqus ._v%Hd^ibn/ ^ difficult^ j . 

^ ^anjd tKe relevance of Xtems were inc^^ed iir ^ tlije calcul^ion of the _st^dard.^ 



■of the Ebel AtethodologV-E^pJ 




Ebel mdragS' - was used f br "fee current <Sor^ Bat^ry 



n theTThree N difficulty dimensions ^st^rit 



:evi^* 




■1 



■ Q -T' 




;• ,12 



ance^dim^hs i&ns ^fcfcss ent £ ai $ :^?import ant ^ ; ' hqSp vefcy; 




.Kara 




1 .' ■- 

Item Re Tftvattr^-V ; ■ •„ , 



Essential 



mi 



■ .0 



* 5 - Uot Very Important - 

J;- ,.-"*!-' • -at .. U|1 \ * : 



1 :'.*^$f$3t? 



,Fflui&- 3: EBEE' : DEeiSieN MTRIX 



: ; important). . The^ reciting 3 x *3 ma 
r ' vj^: Judges werfe ^pres ented ^ith twb 

Answer" : ^tidic^at e 




tg/circli^an M ; i 



/rpteet ^is^f -¥^t dif^jil^r l^ 




«(easyi moderate|l 



: f ating. . Af teir.^pK^; difficulty and relevance of all items we're ratedj a sec- 
bnd fprm was completed by the judges - The secbh^form was used to indicate 

■ - \ ■ ' i : ; - \ ' -• \ : . : - ■ ; • -• • ■• • : • : : • , • - 

fl& ^the;f^^ that, the judge expected a minimally knowledgeable 



teapfier /candidate to answer correctly in each, difficulty/relevance category 
£e.g. 3 easy / ess ent i a 1 , moderate/ important ;* etc.)- A." rating of 0 'percent 
would mean t^hat a minimally know ledge able examinee would answer none of the 



items in the category correctly, whereas/ ft 100 percent > rating 



that all items in ^^^xategory woulcf be answered torrectly. 



tiye items .^inaffe 




> mean 



Description of tfij ^ssay Methodology 



Wfiile thei^i^p^l method seemed we li- suited to the ratlng^pf ^the objec- 



\ method v^ould fia^er to 



in the Core Battery, it was clear, that.: a different 

employed for the essay subtest of the Writing Test. . >' 



Ii^fhis case, it? did not seem feasible to ask judges to read the t^st 4Tem 
.-^pftone a^ e^ay i respbnse ^ '.; It was also q 

^noj possible for judges ta .determine an acceptable ; scof^ . wittipuj seeing 
^cored^ essays. * For thatjreasprVv it was deteriifi^(sd^t^iat' jix<lges/ would require 
: # ^;accessf tc^ actual l essays^ra^^eij- b^ 'examinees tSciJlg the dore Battery in j • 
^Order to determine what . constituted ^minimal writing performance (pilot test 




\^ ^ssaj^s were not availa^l^) 

•Y # Following the sfall-. .1982, administration arid ? coring of the Core . Bat- \ 
^ tery^ , a sample, ©f^ 60 e^ssayl was dfrnm" from the set of essa^ written by 
; L^uisi^gr^xaminees^^ ^est^ing Service ^ -Esisays ;Were 




mm- 



!Eac^ e^$iy ^^eived a "score' between jO^aSadi.. 12, .'' inclusive.. ^ine^_the« ^Ebiltg\ z : / 



The sample was drawn to* approximate t^^^^f^a^o^^f^s^Bf scores in* the : 



State. The writing panelists returned 



Baton Rouge" W : f "ebia^^i- 1983 ^-tb - 



read the essays. Both Content Review aHd Know ledge;- Est ii^^p^- panelists 



'rated the k essays , assigning each A' rating of acceptable, unacc^f&ble, or. 




javerage score assigned. to thos.e papers rated as borderline 
e^chi judge. The mean of these averages was taken as the , 



t on the jess ay .subtest . 



# 



. KNOWLEDGE ESTIMATION PRO€E$S.;^v, . 

... *.y y 'M*Bfi. 

^ v Purpose of 



J - The .StE ^serves as. ; a ce ft i t^^t ion , instrument in>* > Louis i ang . • ^ .' • Indi v^dur 
als-. wishing to teach in kpuisiana ' must , in' addition tb completitig a '•t^ai^SVv :: 'i 



Education program* tajte the NTE pd- r« 



exceed established 



*X : . * leve Is . ;^^i^mbs t prospective teachers , ,-t 
monN Examinations and a tgst sbecif ic to 



t t^ife has meant 'taking both the Com- , 
^t s Specific to each candidate's major- .f^eld (an . * 
Area Examination) . A combined Arc 



e ^immon Examinati" 



dd ebmmoh Examinations Total 
score h^' been used to detepn^je certification** s With . -thye; replacement of the , 

tery, existing certification, r standards 

...y^rl ^ v - :":ij;:.-jr : /. • ; •• " ' 'O ■ - : , 

dm^ijxatibrir o£ th^'/iGp^e Battery mpdifJLe 
in'.$he^ ceftificaticS^decisibn. (See 
[fepter V for a discus ^ v . J :. ■ 





'Since^ the Qorfe Battery scbre$ ^pe 1 \. Op? be , 
it. ^s n^qes^ary to ^^tal 





in the certification- 

a , pfexf pfmanc^ standard. This sstan-, 

* i ■ * ■. '* .■• 7 .-. 



4/ . 
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dard will represent the score at whicli an individual is judged to have suf- 
ficient knowledge to begin a teaching career in Louisiana. The Knowledge 
Estimation panels were formed to provide information, to ^assist in determin- 
ing that standard- ".-i , : • 

"Panelists "were selected to evaluate items in one of eight subject 
areas of * the 3 tests (Professional Knowledge, Mathematics, Science, Social 

Studies tit erature/Fine Arts, Reading, Listening, and Writing). Each par- 

\ , • • " * ■ ■ 

^ ; • .;_ ' . . " - . - K - - - - _. • 

.ticipating panelist rfeViewed test items from the content areai to which he or . 

. • ■ __ . . • ■ .» •• • > - - « * - 

she was nominated. A description of the selection^of panelists can be found 
in Chapter II of this report- While Knowledge Estimation judges, performed 
only one conbeptual task r there Were two major steps in that task. 



Formulation of, a Referehce v Grouag gs^ ; f ^ 

Members of the Knowledge Estimation panjels were required to develop a 
hypothetical reference group in order to make specif ic judgments about the 
knowledge possessed by members of that group. 

First, the judges were asked to refer * primarily to those graduates who 
.were likely to become teachers. *If judges were familiar with their stu- 
dents 1 6^reer plans , they were instructed to prefer only to students who were 
planning to pursue teaching careers. Panelists were directed to refer to 
graduates in all teacher education fields who intended to teach in the ele- 

mentary or secondary grades. ; ^ . — — -~~T 

Second, judges were asked to refer only t^jthbs'e" graduates who were 



just minimally knowledgeable. ^^inl^aTly^ knowledgeable 11 was defined as hayr 
irig the minimal a^imt^of" academic knowledge to a) complete the college pro- 



er|c * w§ 



\ 
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gram required for certification •' in ^1&e-; State? ■^^fw^Bafcli effectively. This 
task was closely related to the practical ^Xp^r5§itc^s ;6Jf ! the paihei members: 

: _ ;._ _ 4 1^ X "J- ^ : ' ' / : 

and thus was likely to be performed with reasofi^b'^^jd^ife^acy.- An assessments 
of minimal knowledge was a- task that every f ^W^7 ^k^J&er * performed , *at ' 

-least in part,- every time he> Or she wrote and 's&gd&d d^amlnatiohs for stu-< 

• V- ' - - • i n^^"'"^ i* : *f ' " •' 

cfents enrolled in college courses or evaluated , a v§tiia^tv^teacher ! s perform- 



5; 



ance. The dividing line between a. minimally pass if^^P 1 ^^'^ %i 1 grade 



must be established, by a faculty member In 'designing '.^^^ : ^^^^e^^^to pre- 
clude setting" the level of difficulty of the' quest^S':^; hxg^that/ it 
excludes the. minimally qualified student from -demons t^at^^ V the \le^M of 
knowledge he or she " has. * That* dividing line must be ; a^sessWSr again in it ad - 
ing examinations; because the delivery of a failing gracie ha^^siioii important 



consequences , most conscientious faculty members, paw ~- great ^attentieirk to, 



their conception of. tthat the minimally qualified student ^Koii^Se: r -abJ'e^feS^ 
do to achieve ..d/ passing grade* on" their examinations ,The >Knqii$s 



:■ -a 

tion Panel members were asked to draw upon this knowledge in -setting the * 1 

■ . : ■ • ; V -■ '. ; m^>- ". ' ■ . : .: .'V- 

dividing line between low passing and failing work, and tb -apply their, con- ' 
ception of the ^minimally * knowledgeable student to test questions written by j_ 
' others. ' \. . 4 — - — f - . . V 

Since the st^andard'^slSti^ was a difficult * andtilrifamiliar one 

to most Knowledge Estimation panelffets, a half -day training session preceded 
^he- -rating tasK. Tn^ purpose "of the^training session was to familiarize 

judges with the tasks to be performed Sid to help e^h judge define ^for him- 

, • • .-■->.•■- ■ ^ ' - . ' 

self or herself the "minimally knowledgeable" 'teacher candidate. -; k : descrip- 



' ■ .... . •. • 

tion of that training is provided later in the chapter.''' 



9 
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Fbrinu i a tiofl oC the Estimates \ v 

Iii the development ^bf the study desist, it was anticipate^ that many 



faculty members would riot, have taught all of . the courses in whifch the con- * ^ j 
tent of the test items wpuld have" been covered. The ^<&i4^, ; ^tu<^^ 'h.T^k^^- - - 
for* example, contains content from six separate disc 




* assemblies, the judges were sent material' that fehcouraged t^^tbv'consuU: 



with local sources of, information regarding the curriculum : ft^iii|iSs|itu- •: 
jt trion; in which they taught,' such as college catalogs , specialists in curricij*^ ^ 

• , ... ...„■. _ _. _■ ■ ■ - ... . . 

lum planning, or other -avail able source^ before attending *the panel ass em- :< -V&% 
-■ ■ . « ■. . - • • V.. ; • :\'^ . v?:^?- , 

j V_V. v, ; * - ■. >: :. .^V^ttft 

bjies. They- were also told that they might wish to talk with colleagues who. / ' 

- had taught specific courses that they themselvesv had not been called upon to 

/►'teach. An overview of the panelists' tasks is included in Appendix IV-B. 

• Judges * were instructed to base their predictions of percentage scores 

8 in each diffici^ty/ reliance,- cell on the probability of" correct responses by 

the minimally knowledgeable te^ vaiidatioii study 

<j w — * — - — ~ > *. • . 

of tl^e NTE in Louisiana, judges had been asked \ ^o T m^ke estimates about 



knowl edge^rather_ than cor-rect— respons es . This was , done to; free judges from 
the responsibility of attempting to determine the extent to w.fiich minimfelly 

Nr know ledge able students would guess the correct answer. Since eommbn Exami- ^ 

. /, . . , ■ . _ . ~ ■ .' ■ . \ 

nation scores "were .corrected for guessing, this was an appropriate proce- 

"., • . 

* - __ j, ■ - ■_ _ . - : . * • ; _ - - , - ■_. •• ■ /■ 

.dure . Scores from the Core " Battery v however ,* are^ not corrected for guess-. 

' y ■- . • ■ -.' " ' ',' ' : • " ; " ' ; ■' 

in'g. Therefore, judges participating in the current study were directed to 
make estimates conS^rning the percentage of items the minimally kiibwledge- 
; able teacher^ candidate would answer correctly. 

The judges were asked to make estimates for all items except those for 

„ ---'-.*- . / ■ ... ' 

'■■ v^liich their experience provided them with no has is for making a judgment. 

_^ • , L '_ £■ : ^- - __Q —L . 

In these cases they . were instructed not to respond to that, item. I 

o - -' ' ' ' ' t ' * . : 
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Knowledge Estimation 

/ . Iil order to familiarize judges with the rating task they were to per- 
form j, a training session w as h eld the afternoon^before the actual' rating 
session was to occur; The training session was organized into three meet- 
\, ings. ; . A 

■ ■■ First, all Knowledge Estimation Panel members convened , in a single 
large-group session. General issues were addressed ai|d an overview of the 
tasks to be performed was presented in this meeting. 

—^Folloying the ^neral session,, Mixed Content (MC) mini-sessions were 
. held: Each of the ''eight MC mini-sessions contained jSaLiielists from a*" variety. 
' of subject areas. The purpose of the MC sessions was ■ to stimulate discus - 
Hsion of the concept of the "minimally knowledgeable teacher candidate." 
; J^Each, panelist received a copy of a handout describing the. minimally knowl- 
: e&ge ab 1 e teacher candidate (Appendix I V-C) . The panelists then participated 
in ; their interpretations of the description in order to 



- -clarify ^ach ^ge 1 ^ perception of the minimally knowledgeable teacher can- 
didate-.<ji;'13i6 di^u^.s,ii^ns covered curriculum issues and broad skill areas in 
addition . tp' : -;|the •d^Jili^n of the minimally knowledgeable' teacher candidate, 
provided; in the handoiilf^| l; ^Panelists were told that the purpose of the dls- 
cuss ion was ribt to reach /consensus , on , what constituted minimal knowledge^, 

vbtit rather to aid each panelist in clarifying aind stabilizing his or her "o* 



* conception, of the minimally knowledgeable' teacher candidate. In other \ 
words l each panelist made a set "of independent Judgments. Before doing so, 
however, a training forum was provided to develop a stable' concept of the 

^minimally knowledgeable teadher candidate in each -judge f ,s mind. -The MC ses- 

*-"<v - ' ' ' : '•' ■'. • 

^s^ions lasted approximately brie hour. . 



i 
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: *i FpllowMgvthe -MC' r mini-sessions , Same Content (SC) sessions were held. 
Participants in^each SC session were nominated:; to serve in 'evaluating tfre 
zs anift . A Cote Batti^^TeatT^SC sessionsHreg^— with > '-a brjgf \ revi ^ r^f^^ ^^C 



sessions . . Fbllbwitlg? the initial discussion 3 judges were provided with ' saffir 
pie item sets, forms ^ and directions for- completing the Knowledge Estimation 
rating task (Appendixes IV-D tind IV-E contain sample; forms and directions) . \ 
Seven of the panels, were provided with \ 10 sample items. The Professional; 
Knowledge ^anel received 20 sample- items since their actual rating task ;was . 
to m be significantly- longer than that of tAe. others panels.- Half of the 
items were xated, and these ratings were tabulated and discussed. '.Then the 
remaining half of the items were rated, tabulated, and' ' discussed. Judges' 
then predicted the proportion of items in each category that would . be 
♦answered correctly by" the minimally knowledgeable teacher, candidate,. Fol- 
lowing tljje ratfng of the sample items, panelists entered their results into 
a know ledge estimat ion trainin g w orksheet (Appendix . IV-F) to demonstrate how' 

the -ratings would later be used-->to determine qualifying scores . These , 

" - ■■; .: Y- Y 'I < .- : " . " ; , ' 1 1 - : '/ 

results wei;e also compiled and discussed. 

■ • . ■ .. \ ...... . . rf/ ' • ■ - .•: : 

Description of Ratings 




Item ratings were conducted on the morning following the training ses- 
sion . Ratings were counterbalanced by both form and ta^jk. That is , half of 
the judges rated Form B first, and^the others rated Form A first; half rated 
item difficulty first, and half, rated item** relevance first. Items that were 
common to both forms were deleted from, the Form- B packets f During data pro- 
cessing, ratings of common items, (those "that appeared on both Form A and 
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•Form 'S^psr^^inciuded' i$;"the to4 lysis of each test form to develop compara 



ble JratifagsV lar^/gehe^^ session was held for HbKe rating ,t ask. Pane-: 



^i^fcsr^vtfEra^ encouraged not v tp sit with one aiioLher, 




and Vjudge,s were instructed ^ot f:o confer among thp.msp.lvp.s . : , . « ; 

- • -ify?,^ * At the . r t ilia of ythe^ia£irigs , ■ only one fo^rm of the Listening Teit was 
.' ;-. J';^^^Ble ±3' fifi^i fprnti/f While" a|typed version of the final form was avaJil- 
{^al^i^v it was f elt/ tha^t > judges should hear the Listening. T^st rather than 

' r 'rea^ it': ; Fpr' that ; r^ason/^ only the one ^ complete. ' form of the Listening Test 

•yU * ' : I < /; : );P : P ■ : , * • ■ •' * _ - '_ ' > jj 

&%±ix /tye .Dctabeti'';; 1982, session. Panelists were invited to return 

' / '•* . ' ' ■ ' '/ V ■ ■ • ' • * 

iff; February, 1983, to' rate the 'second Listening Test form. 




-^Seyen judges / returned for. J this, rating session. At that time, they listened 

^"'i'-'- / / '.j_'-J %'L:-/\::-j';: J _L '- : J 1 ' • • 

::-Mo the first form of v't-he- test and reviewed their ratings of that form to 

' -<- J\.jj:£±::. • . > •■: : 1 .. - u \ : ... 

>"■ r$- establish the framework from which they made thefir Form A ratings. Fol- 
lowing the' review, each judge rated Form B in the same manner- in which he or 
she had fated the first form'. 



Since the es^ay validation methodology required a sample : of examinees 
essays, Ratings of the essay subtest could hot be conducted* during the large 
rating session in October. Instead, Writing panelists reconvened in Febru- 
arys 1983, to rate the essays, A sample of 66 essays produced by Louisiana 



° examinees wa? chosen to approximate the overall distribution, of essays in 
thk State. Panelists from both the Content Review and the' Knowledge Estima- 
tion Writing Panels were Jjivited to participate o iri the ratings. Only essays, 
from the fall, 1982, administration of the Core Battery* were available for 
rating. The time constraints for completion of the study did not allow for 



/rating the spring, 1983, essay form. 



eric 



As explained previously, eight panels of judges (one for .each test in 
the Core Battery) were formed to provide Ratings. These panels were formed 
in the^ areas, of Professional Knowledge , Mathematics , Science, Literature/ 
Fine Arts > SoQ^< Studies, Writing, Reading, fed; Listening. - ' NTE .Core- Bat- 
tery - scores are not reported at this- level of detail, however ; so following 
the *det eirminat ion ; "of the panelists' standards for each test,; test scores 
were aggregated £0 form scores for " each module, , Mathematics^ ^Science, 
Social Studies, and Literature/Fine ^ Arts were combined to form the General 
Knowledge module. Writing (objective) , Writing (essay) , Reading, and Lis- 
tening were included in the Communication Skills module. Stores in 'this 
module are weighted so that each test -(Listening, Reading, and Writing) have 
••• equal weight ..'Within the -Writing Test the objective and essay subtests are 
weighted equal ly .' ; Profess ional Knowledge is a complete module.: / V 

t ' - y \ 

At the" time that panelists ratings were aggregated to determine stan- 
dards , data from the' Content Review Panels' ratings were merged, with' the 
Knowledge Estimation data..; a Items v judged content appropriate (see Chapter 
III) were given a weight of i . 0 . Content inappropriate > items were weighted 
6.6. Knowledge Estimation rat ingi; were, multiplied by their content appro- 
priateness weight during data processing. Thus , content appropriate items 

- ' ' ' • \i ■ ■ ■ . ■ - • . 

'"*..*; » f - » ' ~ . .... ■■ . * - 

were included in the determination of the standa^d> : but contend *dttapprppri- 
jate items were dropped from the calculations. * : "v: ' * ' 



KNOWLEDGE ESTIMATION RESULTS 

Essay Ratings : , . ' > ' ,. / - yv 



Tafe^e < 20 presents information concerning the distributions of all 
essay scores*' in Louisiana for the November, 1982;; "administration of the Core 
jattery and of the- sample of papers selected for ^review. As* can be, seen in- 
Table 20 i a slight- oversampling was required in the' tails of the distribu- 
tion (thp. high and low ends) to provide representation at ,all score levels . 
At least two essays were sampled at each^scbre .^oint . fable" 21 includes £he* * 
average number of essays placed in each of the three rating catgories by 
^Content Review judges, Knowledge Estimation judges, and the total group. It 
can be seen front the fable that there was general agreement between the two 
types of panels in the percent of essays 'assigned each category. Consent 
Review panelists rated somewhat more papers as "Borderline, 11 Vhereas Knowl- \ 

edge Estimation, judges piacect more papers in the. "Acceptable" category . 

; -s ■' " ■ ■ ■ ■ ; . • . .. ." - : ■ * ' ; 

. _ j n f < j rmat i on - concerning * the average scores . assigned to borderline ; 

• --. "*' ■ \t "Sr. . • ■ •• . ' .- '. ■ 

papers is presented in Tab 16 22. Both the mean and median are reported in 
.Table 22.' The median i^ inciud^i, since it Ms ;a more stable statistic titan 
the mean . In the* case in which one judge provides rat-ings that vary widely 
from those of other judges, the mean will be more affected by such rating 
. than \the median'. Insj^action of Table 22 indicates that this, was not^E^e ' 
case. The median was 6,9.3, and tiie mean was 6.91. fherefore, the mean was 
used in all -further calculations since it incorporates more informaAon than 
\ the median. -Based on the results shown in Table 21, it. was decided to use 

• results from the total panel in the calculation of the estimated score for 

■ j •_'»_•■ ■ - • . ■ . . * '* i _^ _ 

the minimally knowledgeable teacher candidate". Note that Table's- 21 and 22 
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TABLE 20 

. » " * " " 



PROPORTION OF LOUISIANA ESSAYS RECEIVING EACH POSSIBEE SCORE 
NTE CORE BATTERY ADMINISTRATION, NOVEMBER, 1982 




12 
11 
10 
9 
8 

' . - 7: 
,"8 
5 
4 
3 
2 



I* 0. 003. 
i 0:0l4 : " 
0.058' 

ens: 

0.224 
0.196 
0.183 
. 0.094 
0.069 • 
0. 02Qs> 
0.012 ■ 



i 0 . 033 
0.033* 
0.067 
0 . 100 
0.200 
0.183 
. O .'167- v 
0.083 
0.067 
0.033 ' 

-frr033^ 



. - : ' . _■ TABLE 21 =' "± 

' ■ ; f :' - ■. . : ' : ; ' ■ -.'i 
• average percent lot essays classified* in each 

Judgment category by-, nte writing, panel members • . 



Panel 



Acceptable ' Unacceptable Borderline 



Content Review 
Knowledge Estimation 
<r : ' Total Panel -; ! . : ' 



42.50 
48.89 
44.63 



. 24.44" 
25. £8 
- -'7*24. 72 



33.06 •, - ; , •• 
.25 . 83 i f - , \ 
30^:65 " - 



present the data by the judges 1 original ta^k assignments (Content keyi^w. c^r. 
Knowledge ^timatiott) r , Half-panel results ^BifcjKn^^ 
sented later -in thikV c^kpter . y \f *J '•',<. ' .?V 'S-.-^*y'&*'\ .■' " 



V \ ^ ;s ■ • TABLE 22* t , . ' • , f , Y ' ' 



MEDIAN ANF MEAN dF S60RES ESTABLISHED BY JUDGES AS 



, REPRKENTI^e-BeRDEB^m-ESSAYS- 



Panel 1 v Median of Judgejs t- -R^i^gs ; Mean pf 'Judges 1 Ratings 



: Content Review 6.?^ - / * 7,04 

Knowledge Estajnktioh 6 • 65 ''■'/•■ ■ 6.65 

Total Panel ... '[..■' 6.93 '■, 6.91 

; ; ' - - : ' _ „ ■■■ - • ^ , , 

Ebel Res u Its •' ,v , • 

* Results of .the Knowledge 'Estimation -Panel; ratings of obj ective test 
■ items (the Ebel xatipg^) ate presented in this section. three tables are 

^ provided. Table 23A includes the distribution, of relevance ratings by test. 

•• ... . ,-. • ; . ^ ' - -■ ■ - . -, . ' ' ■ 

The average percent and number of item judgments classified as measuring 

* "^ssential , 11 "important , " or' "hot very : . important 11 knowledge is reported for * 

each -form -^qf e^ch. test. >It - can be seen frdm . Table 23 Al that ■ j uciges t elided to : .. 

/rate - items, as measuring either "essential" or ,l not very important 11 knowledge o 

/ more often:- than they pirated-, items as "important . 11 Additionally, judgments 

. were more likely to be in the "not very important" category' than any . other . 

.s*'W^le.. ; : €h^^|s^ results were not expected, they are not altogether surprising* 

V /Judges *t?er^ f instructed to classify the. knowledge required to answer . eadh 

/ item ^brebtiy as^esisjsnti^l , " "import^tv\ ^,Vh6t very important." It is 

j likely 1 thati r at her than e valuat ing the taabw ledg^ required to answet an item •' ** 

^l^e^tly, j5dges raped the overall relevance' "of each item. ^ In ? isplationV 

th^ specif ic content , cjf a v given J item (rather than the too'wle&^e it repre- ' 

sents) '. : l"s hot likely to. jbe; considered essential . in . light of the higher 

thdii ^xpected percehtige of ratings in^the "not Verjr imppf^a^ 

.-. /, . i ;v\': . * ■■■ ^ t . ?/ J ,, n k. . T JV • f , v « v = \ » 

»^ v - r/'-'v--/:*^- i, * -> ; . ■ /'""'"^V- ' ; : .•*•-'- * - ■■ - .... ,-. 
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- further analyses tfe're conducted; The results appear in Table 23B> which 

r: <i,,:, * < ■ • : ' - ' - ~ - - ; ', l 4 : t 

presents the number and percent of items rated Essential or Important by a 



i majority of judges . If over half of the judges for a given test rated an 
item as either important or essential , that item was included in Table 23B. 
It is "apparent that most of the items in the Gore Battery were not consid- 
ered unimportant using this criterion, in contrast to the percent of judg- 
ments- tallied in Table~1£3A. Thus while there was an appreciable number of 
• ■■ - ■ / . . ■ ... : -< - • . . . ■• • v • ■ . »■ i . • . 

"not very important" judgments , they tended to be spread across items; very 
few items received, this v rating by ,a majority of judges . 

Table 25 follows ^ ttie s|n^e format as Table 23A and 'reports- the. diffi- , 
, cii Ity 4 ratings . About . one-third of the item judgments across tests were v - 
classified as "easy.": In six^of > the eight tests more than 70 percent of the 
. j udgme^t s were either "easy"^ or "moderate" on both Forms A and ' B . For Form 
>the v Listening Test , .65. 5 percent bf'V the item judgment wer.e "easy" or 
^'mc^^at^'V an^ for Form 3 of : the Literature/Fine Arts Test, 62.0 percent of 



the ratings fell in these categories. 



Calculation of Judges' Standards 

There were several., steps in the analysis of judges 1 ratings to calcu- 
■|ate an expect ed Minimum score. First, data from, the Content Review Panels 

•' :■. ■' ' ,. : V-* .-J*"- ' . - • .... . •• - . ■. ■ 

: _ - ■)— - •' _ ; L^5*r^"- •_ , - '_ !_■■* : 

were merged^, with the Knowledge Estimation, ratings prpvided^ by each judge • 
TOe ratings oi£ items judged to be Content . inappropriate Were deleted from 
. Xf^ rating, of each item 

.;, ( easy/ess ^litial , mbderate/essential , etc. ) was replaced by the - percent of 
" ' it^fis IS Sfi& ^ j&e . answered correctly '-. v ;. 

• k byy£h# niihiSnaiiy ; "knowi^pable teacher candidate . •' litis percentage was then " 
,%6hveittd to a proportion indicating the probability *tfiat the min .'j- 
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MEAN NUMBER AND PRO! 



>CED IN EACH 



RELEVANCE CATEGORY BY KNOWLEDGE- ESTIMATION PANEL MEMBERS 



Test Md 

\ -;.f 

Form 



X- ;"/' Not ? Very Important 
\ , \\ t Number . Percent 3 Number Percent* ; . Number Percent 



Essential 



Important 



Professional 



Form A 
Form B 

Mathematics 
.. Form A 
Form B 



30 . 7 
32.6 



10. &, 
U.S. 



29.5 

32.0 



4X2 
0 



17.4 
16.3 



3.2 
2 . 7 



16.8 
15.9. 



12.8 
10.8 



56.5 
56.1 



10.9 
10.7 



•54.3 
p} 55 . 0 



43.6 
42,8 



Science 
Form A, 
Form B 



8.5 
7.3 



28.2 
24.2 



6.9. 
6.8 



■23.0 
22 . 7 



14.5 
15.8 



48.2 
52.7 



Social Studies [j . : L 

Form A 8.0 ,. 26.7 

Form B 10.1 33.6 

Literature and ; 
F.ine Arts 

Form A 10.2 29.0 

Form B' 9.1 -26 . 7 



'5. 



7.4 

6-3 



6.8 
9.1 



24.7 
21.1 



19.3 
26.7 



14. 6 
13.6 



18.0 
16.8 



48.6 
.45.3 



51.4 
49.3 



Form A 11.0 ' 36.7 

. : Form B ; 11.8 V 39.2 

Listening - 

;•• .Form A 18.8 46.9 

,. Form B 20 .6 ' 50. 0 



2.7 
2.3 



7.9,' 
5.4 « 



8.9 
7.5 



19.7 
13.6 



16.3 

i .15.9. 



13.1 
8 . 2 



54.4- 
53.1 



32 .8 

20:6- 



Writing (Objective) 

Form A 15.4 34. 1 

Form B 15.4 34. 1, 



6 .5 
5.2 



14.5 
11.5 



22 i 9 
23.7 



•50.9 
52. 7 



■■■ ■ a * ".-."v- >7U-^-. • . . . ' • ; ; . . • : , • :- ....... . 

. ^Pe^Biitag^s • : «a^.^ot ;;-tbt il §0 ;^e-araef 6f • xoundingi -aud no^r es,pons e , :.' . ;. : 



ERIC 



37 



TABLE 23B 

NUMBER AND PERCENT OF ITEMS RATED ESSENTIAL OR IfiPORTANT 
BY A MAJORITY OF JUDGES FOR EACH TEST FORM AND * 
. ' " THE AVERAGE PERCENT OF THE TWO TEST FORMS 



<* : 


FORM 


A . 

» • . 


FORM B 


■; >t" ' •■ 

- '/ ; 


AVERAGE 


* 


Number 


Percent^ ^ 


Number 


. . f ,_• 

Percent 


Percent 


' ' ' • ^> . :■■ ". .. . ' f 


of ^ 


of 


qf ; 


# •' of / . ' 


bf 




Items ' 


Items 


Items 


Items', 


Items 


Professional Knowledge 


'99 


95.2 


97 


95.1 


95.1 


Test/Module 












'Mathematics Test 


24 


96.0 


24 : 


96.0 


96.0 


Science Test 


, 26 


86.7 


28 


"* 93.3 - 


90.0 


Social Studies Test 


#26 


86 .7 ■ 


27 


' V 90.0* 


88.3 


Literature/Fine Arts Test 


51 


88.6 


28 


82.4 


85.5 


General Knowledge Module 


107 ' 


89.2 


. 107 


.89.9 


89.5 


Reading Test : f 


30 


100.0 


30 


100.0 


100.0. 


Listening Test 


39 


97.5 


40 : 


.100.0 


98.7 


Writing Test (Dbjective)- 


44 


97.8 


44 


97.8 


97.9 


Cpmmunication Skills Module 


113 


98.3 


114 


99.1 


98.7 * 



knowledgeable teacher candidate would answer that item correctly. As an 
example,! supposi 1 an item had been jiidged to be moderately difficult and 
import ant\, and that the judge predicted that 50 percent of siifch items would 
be answered correctly by the minimally knowledgeable teacher candidate. The 

_ • '. ;• • * __ v;\ _ ' -_l '_ _ ._ ■ 

probability of a minimally knowledgeable teacher candidate -answering an item 
in this category correctly would be .50. The sum of such probabilities for 
all items in the test is the predicted raw s~tore . (the number of items 
answered correctly) , of that judge. The plwicted raw scores were averaged 
across judges for each test. These average test scores were finally trans- 



formed to scale \ scores for the three modules of the Core Battery using 
transformation formulas provided by ' Educational Testing Service. The 
resulting scaled score standards are presented in Table 25 i. ^ 
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_ : TABLE 24 >■ 

MEAN NUMBER. AND PROPORTION' OF ITEM JUDGMENTS PLACED IN EACH 
DIFFICULTY CATEGORY BY KNOWLEDGE ESTIMATION PANEL MEMBERS 



TeSt and 
Form 



"Easy 
Number . Percent 



a 



'Moderate 



Percent 



Hard - 
Number Percent 



'Professional 
Knowledge s.y 

Form A:. ' ..'27.^S: : ' 



Form B ,.J«;J 

Mathematics 
Form A 
Form B 



Mi 



-26 i 4 



2g.9$^ 28.4 



36.4 
36.0 



9.1 
.9.0 



48i 4 



45.4 



10.0 
10.6 



;*c.46 .5 
i; :'44.5 



40,0 
42.4 



'; ; .r-'27'.6^'^:'.v^7>OVv 



5.8 
5. 3 



23.2 

2H2: 



Science 
Form A 
;.. Form B 



11.5 
8.8 



38.5 
29.4 



12.3 
13.2 



40.9 
43.9 



6.0 
7.9 



20.0 
26.4 



Social Studies ' 

Form A 12.7 42.2 10.6 35.3 6.8 * 22.5 

Form B 13.9 46 . 4 10.7 35.6 5.4 18.1 



Literature and 
Fine Arts 

Form A . 11.8 
Fori: B . 8.5 

Reading 
., Form A 9.4 
Form B 9.3 



33.6 
25.0 



31.4 
31.1 



14/6 
12, .6 



14.3 
1"5.6 



41.7 
37.0 



47.8 
51.9 



8.6 
12.9 



6.3 
5.0 



24.5 
38.0 



20.8 
16.7 



Form A 
Form B 



12.9 
11.4 



32.2 

28.6: 



18.6 
14.8 



46.4 
36.9 



8.4 
6.2 



21.1 
15.6 



Writing (Objective) ■ 

Form A 10.9 24.2 20.7 46.1 13.3 29.5 

Form B 12.9 28.7 21.1 ' 46.9 . 10.5 23.4 



Percentages , may not total 100 because of rounding and nonresponse. 
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r The results for Form A and Form B are similar for' the Professional 

~ •' "... . ,_ • « - '_ _' - __■ ■_ • l _ . ■_ 

Knowledge arid General Knowledge poduies, with a ran^e of no more than three 
scaled points between Form A and Form B* However ? the range between Form A 

and Form B of the Communica t ion Skills mo dule was greater. This was -largely 

(jp-- ■ — ■— — ^ - * r — '- _______ _. .. . • _• .. 

due to differences ,in the number of judges rating the two forms of the Lis- 

•<» ; " <- - " •■ -2' _ : " . 

tening Test.. One judge rated most of the items * on" both forms as; "riot very 
important, 11, The- impact of this judge's ratings Was gteSter on .Farm B tfian' 
^ortf ft ^ 

dard fpr . t^ . • • V • • ; ./^ -A 



CONSISTENCY OF .RESULTS- FOR KNOWLEDGE ESTIMATION PROCESS 

The half -panel analyses, for Knowledge Estimation' changed from the 
method employ ea the 1978 validation study to match the changes in the 
procedures used f^r, producing Knowledge Estimation ratings . The half -panel / 

analyses presented 'here consisted of four parts: a), a comparison of the 

• - : Jfc'- • • ■ V ■ '. ■ i. . '. . ' ■ * ... . 

:."_ " • •'. TABLE 25 



JSULTS OF APPLICATION OF EBEL METHODOLOGY 
STANDARDS BY NTE CORE BATTERY \lODULE 



4 Module • . .. ; 


Maximum . 
• Score 
Possible s . 


Form A 


Form B 


Average 


« :\ :v. •■■/ . ~ . . . . ; >:\£ 

Professional Knowledge 
Geheral Knowledge 
Communication. Skills / 

■ ■ •. • ' (■ 


690 
690 

690 • •• 


654 
652 
656 


651 

. 650 A 

••. 647 J 


652 

651 'I:''- 

652: .'■ ... 


• 




\ 







half -panels 1 determination of Ebel methodology standards by module and test 
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iv. ' ; 90 
hgs of item difficulty by^ test aiid 
ratings ' of it ©in relevance by test 
and test ;form; and d) a comparison of half-panel ratings of the percent of 

» \ i -- - ...... '.' .; 

items -the minimally knowledgeable, teacher candidate would' be expected to 
answer correctly by test. ..^ 

In s order to apply the Ebei methodology, -the judges gave ratings for 
item difficulty, item relevance, and the percentage of items in each bbmbi-.: 
.. nation b£::: v &£^ 

; ; "candidate would ; answer cprrect iy. • • The half -panel, analyses began with the 
end-product of these judgments and worked ' back to contrast each component 
part across half -pane ls . This procedure sefved two •functions: it presented 
the consistency of results for the overall process and it examined the 
sources of inconsistencies when they appeared. n '■ .,' 



Consistency of Ebel Results v ^ 

The* 'final results from the derivation of the Ebel methodology stan- 
dards by half -panel are shown in Table 26. Ha If -panel estimates of the 
standard scaled score that, could be expected £f the minimally knowledgeable 
teacher candidate are. shown for both Form A and Form B of each module. 

_ . _ .. \ _ _ _ _ _ . _ • _ • _ _ ' _ •;/- . ■ _ _ _ _ _ 

Differences in half -panels estimates ranged from nine points (Form A of 
the ^Communication Skill module) ' to three points (both forms of General 
Knowledge). T-tests of the dif f erences ^between' raw scores assimed by the 
two Half-panels to the eight tests comprising these fijoduies Indicated that 
none of. the differences between half-pbnel ratings were statistically sig- 
. niflbaiit ; - • ' ; . ' 
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TABLE i 26 



RESULTS OF APPLICATION OF EBEL METHODOLOGY 
STANDARDS BY#TE CORE BATTERY MODULE, 
% FORM, AND HALF-PANEL: SCALE SCORES': 



Form A Form B - \ Both Forms 

Panel 1 Panel 2 Panel 1 Panel, 2 Panel 1 Panel 2 



• - -- . -\ n 

♦ Professional Knowledge 651 656 ~ r 649 654 .650 655* 

General Knowledge 650 653 648 7/ 651 . _ 649 ,- 9 652 

Communication Skills 652 ' 661 651 \ 658 652 ' 660 



After determining that the overall results from the Eb&l methodology 



standards were consistent across half-panels, each task in the procedure was 

_•, _. " - -f ■. .;■ ■ \ : 

then examined to determine if this consistency had been maintained through- 
out the standard setting process. Two analyses of the item difficulty rat-y > 
mgs are shown by half -panel in Table 27. These are Pearson product -moment V 
correlations of the half -panels 1 judgments of item difficulty and average 
differences in the half-panels 1 ratings of item difficulty. . 

The Peaxsbn product -moment correlation values between half-panel^jj/dg- 
mehts of item difficulty ranged from .39 (Form A of the listening objective 
subtest) to .73 (Form B of the Mathematics Test). All correlations were 
significant at the p<. 01 "level or greater* and 13 of : the 16 " correlations 
were significant at the level of p<. 001, It, was concluded from t^is test 
that the half -panel ratings for each of the Core Battery tests showed a high 
.degree of association.. 

The differences in: half-panel ratings of -item difficulty could range 
from 0.00 (if the average difficulty ratings were identical for both half- 



\ 
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TABLE 27 



, COMPARISON OF HALF-PANEL RATINGS : :i 
ON ITEM DIFFICULTY BY TEST FORM, - s . 



■■I 



Examinations 


, Form A 
■ •' ■ ■'. Mean ' 
Correlation Between Difference in 
Half-Panel - . Half-Panel t 

E D 

• Judgments on Items Scores on Items 

' .< 


, ; . Form B 

; i.V.-Vj ■ Mean 
Correlation Between Difference in 
t ^ Half-Paiiel '' - Kalf-Panel r : 
' Judgments oaltetfis, ScbreB\% v XteiB8 


Professional Knowledge 
Mathematics ' . ' '•. 


• .67 . , . ' .04, 




Science , • 


- ^ .69; .li 


/ .62* . ' «".01 ■ - 

< ■ .53 \ < .01 ^' " >. . 


Social' Studies , 
Literature/Fine Arts' 




,JJ ' -\ ■ ■ ./ ... ■ ■•4*. , p . * • ../■; 


Reading. . , " f ' 

Writing (Objective) 
c 

Listening 


* .57 ' 
- . 72\ s .33" \' 

; .39 • : " K -;--.a_ 


' ■ .57 ','r--,.. ,31 ■ 



Pearson product-moment correlations are reported. Ail correlations were ^ •'" 
. except for the correlations for Forms A and B for the Listening examination.-. • These two coi^elations'v^- 
(-.02, .08) were not significant.- . • -oV-' ; > . - : ■;. • v : .''"' -i: ■ ^- 

b _'i . .-J"' : --•-- : ' ; r •**i ; «-;Vv.' V V> V" 

The differences reported here were computed by comparing the average difficulty rat'ing .'for items given by • 

Panel 1 and Panel 2, The. range for these differences .'was G -tb 2, since the minimum reievalce rMg Was • 

1.0 and the maximum revelance rating was 3.0." ; ' ' .^V' . : ' ' : O ^ ' 

. Note that half-panel composition- is different for Forms A and B for ^fstening as explained in Chapter IIv/ 

. ; • ... > . .. ; ■ ■. -SO 



panels) to 2.i)0 (if one half-panel gave an average difficulty rating ( of 

1.00* or "easy*" and the other ;. half -panel gave an average difficulty rating 
'■.of 3.66 , or "hard") . The range actually observed for the^e differences was 

f^m .01 (Form B..of the' Science and Social Studies Tests)' to ..33 (Form' A of 
' the Writing Objective •subtest) : The ayerage difference between half-panel 

ratings acrosls all tests was .15. v ■ ■■••'••■'■-'.Vr : 

Similar Comparisons of consistency for item ^ re^yance: -ratingi 
;sfip^ Listening 
-test : .02 for Fbrni A and : 08 for correlations ranged 

froni .35 (Form .B of /Literature/Fine Arts) to. .85 (Form B of Mathematics). 
All correlations i with- the exception- of those for the Listening Test, .were 
*s^^ificant at the p<.05 or greater level.. Eight of the correlations were 
significant at the level of p<.001, indicating a high degree of association 
between ha,lf ^panel ratings of item relevance. • 

The overall correlation values for item relevance are very similar to 
those for item difficulty, with the exception of the item relevance correla- 
ti^ns- -for tfi^Listening Test. Thpse low correlations, noted in the preced- 
irig paragraph* occurred . because one panelist rat^d most of the items in the 
Listening Test as "not very important." The effect of this, panelist's rat- 
ings was more pronounced on Form B, for which there- were, seven judges , than 



on Form N A, for. which there were nine judges. There were ho procedural rea- 

. . ... * / _ , / 

sons for excluding this panelist 1 s judgments, but the ; half -panel correia- 

tions for the Listening. TesV should be interpreted with 6 the understanding" 

that they include one set of opinions that diverged widely from those of the 

other judges. - / 



TABLE 28 



COMPARISON' OF HALF-PANEL 
ON ITEM RELEVANCE BY TEST 



• FORM • 



.« .J 

* 


Form A ; 


, « 

: , ,v. , : Form B : . • ■ ■ 


.Examinations 


.::»;. .. ^Meari , 

Correlation Between Difference in 
\ Half- Panels' a Half Panels' 
Judgments on Items Scores on items 

\ - - — — 1 : 1 


, - " -Mean. . . "I l 
Correlation Between • Difference- in ■ ' 
Half Panels' & ' Half Panels' . • . 
Judgments on Items Scores on Items . 


Professional Knowledge . 
Mathematics t : . 
Science 5 <*> 1^^ 


' .59 ':'"">••■ ',22' 

' . ; \,, ; ',',59 v ■■: *,09' ' ' 


•••••>;*'-:<:r -V'- :r-:.^v^' i -r 
"■' ,.53 •*,.' ' 23 •■• ■':< 

; 1 AS ■. : ,35 . »:,:. 


Social Studies 






Literature/Fine Arts 


. .68 ,08 


' " .35 .04 


Reading . 


.47 ,03 ' 


, .< * 1 it ■ ■ ■ ■ ' ' . ■■' ' • 

• .37 * . _ ,06^ ■ 


Writing (Objective) . 

• V - 
c 

Listening 


.53 . ,14 - 
-.02 •' ,42" • 

» 


:':^ : 'm-?''- : r /f ^' d ■' : \ 



Pearson product-moment' correlations are reported. All correlations weft significant at the £ < ,05 level, 
except for the correlations for Forms A and B for the Listening examination, These two correlations 
(-.02, .08) were not significant. 



The differences reported here were computed by comparing the average relevance rating for items given by ' 
Panel 1 and Panel 2„. The range for these differences was 0 to 2, since. the minimum relevance rating was- 
1,0 and the maximum revelance rating was 3.0. - ' 



c 



Note that half-panel composition is different for Forms A and B for Listening as explained in Chapter II; 



• lie 
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- . •■ . -■■ •. •••.•i-^.- ' •:' VV- • . V- . : :*S .•" . : 95 ^ • 

/' ' * ' ■ • •■ •*• .-• : - ;v v :-. ; ■ ' ? ■■V- KV. 

The difference between haif-panei ratings o£ /i fe m reliance could- : • 

range from 0:00 to 2.00, ,as ttiey could for :it^ 

0.00 would mean that the two half rpatiSls made idfeiiticii Average i^tift^s* ' fbfcv 
item reievanca. A'diffeieiteji' of l 2>0t) would '< mean tia£ oiie^rhalf-pknel jijdg%d • 
*all items to; be "not V^ery impb and the -other half ^pajiel judged 

r A:all items to be "ess^t ial;"; (3 . 00 )„ 'Qie observed range b ^6r tfie^e differ -S 

; entes was highest for tha - Listening- Test : .42 for- F5rm A and ;6l^ for Fbrii B . K 

•' ---Vi->.- - / • - .-. ••••• \ ^ . -4 i •..•.} . ^ - ' ■:• '. % - 

Affibng the gbther 'tests', the ^aiige w p as- frbjji ; ^03 , (for. Foirm A bf the Reading * ' :% [ 

^Tes.t) to .35 (for Form B of the "Mathematics Test) . Excluding the vEistening. . ; - : 

-• '! 'lr " __' • - . • . ' *' i '- 2 ^: ♦ - : . . •' " . .'• '; , ' . .\M fc':. 

Test£ the average diffefeixce across -the 'remainiiig seven tests was ilS ./ : Ttil$ 

was the same average difference as rthat found fbr f item difficulty/. \V • / \- 

' ; • . The; final- comparison was iiiat between half--p<toel r e§timatBs of the per- r ; ^ T- 

^.r ,, ':l--. . • " ^' ^ ' ■ • ; v." : ^l 1 -.-. ._ ; 

. cent of items the minimally- teiotfiedgeable teacher candidate would answer 
cofreptly. That ijiformatioii is presented in Table 29 the- values f«3r' the ^ 



v half-panel correlatibns were cons is tent ly high , ranging^ f rjom .88 <§n the Lisr- * 
tening Test^tb* 197 on the Professional KnowX^dge ' $nd Literature/Fine Arts v . 
Tests ; .for _ half -'panel 'differences in perceStage^ ^stirn|,tes , the-, vallies cduld v.* 
range frpm,- 0. 00 (cbrpplete ivergige agreement )^ to ,1 .00 • {complete disagreement , 1 



on the average i for all nine categories of difficulty and rdleyaiice) . The ■;* '5 

observed range was from .02 fox the Liteirature/Fine Arts Test to f .20 top the 
r Mdthemat ics Test; The (average difference across tests was - .10. . , 

Consfetency of t Ess^y Results jf^ : •• . ' ' C ^ 0 V V ^ % J 

* ' * _ 'J _ 1_ _ ' J. .■" a. J. "~ ■ ". " _ - - jl! - V r - " . 't» ■ * ^ 1^: ~ ' '[><> ; 

The « Qonsiste^by^meas^xres described, above were/ not appropriate for 'the 
essay T subt est of the Writing Test^be'canse: th^ 



dures used here differed fror&\'thbse employed with the other tests. Consl's- v 
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COMPARISON OF HALF-PANEL RATINGS ON PERCENT OF ITEMS 
THE MINIMALLY BM)WLED.GEABLE -.TEACHER CANDIDATE? WVlft 

-'•'.'•"^^vv,'.., BE EXPECTED TO ANSWER CORRECTLY ; ' " ■ ". 



•«*/• J"*' '"-.i ' • V ' *< '• i' . ««3f 



t\. ■: • - : : • - :v;-, — ■ ;,rr i *: ., ; - ^.f^ 1 -,■ — — — — : — : : — • ■ ^ ; . 

■ '< . • ' . . v ; : V , Gprral^aoipt^BfetWeeiiy : ' ^ 'Diff e^enc^^BtweBii, . <" ";• - 

'V ' ' "'■■s.-- ' _, , ■ * • c-v: 4 • 1 . •• • 

" V: : : V , •' ri ' w-:.;>>\^: " . Half -|>lii^|^ : |u^^3 ' Hal£-Paii£is I: ;^r^^vi : i- > 

::r V ; - ■ ^-v ^'^L^'^sjs; Categories . '•• 

Professional Kiic^l^c^is:: , . : V -9;^;>^/^^:c;X fv; ; . ^|^^^^f^; f ; v*"-'"' 
V ^ ; Mathematics ^- n :^ 'r*^:;;^^-^-S>..:-: : - . .95 " % ■ .^4?;?. " J.." '\ 

Science ">'..- v , V ' ' ;';i ; - : ^V? r < .$^ v - ;:. ; .' .;^ ! v fe- 

. ' Social ; Studies ■X':^;^: V> , Vio'.t-:^*' "' " %/ " ' ' 

>•:> N ^Literature/Fine Art? v, - v-' f -, ;-;*97 •• : *' ' '• V ' ^ . ' 

4 Waiting (6l?j^ct.iye;); ^ , ■' %-:ijf.«:,v .'90^ ^ ^^i'-^ , .04 1 * f ^ 

£ist&iAg v: v . y^U-:'-,;^. s £ £-J-^ ■ : - * . 

v • Ws ^rr^a^bHs report 

' > All correlations W^re signif icaiitr • at-) -the r < -Q1" level. , • ^ 

: * b v i:/ ; • :i " f V ..; : .V- ^ •. . c . . • ) r \, : y '\ . ; > 

: i ; V' These differenced . were computed By ^comparing the 6vfer age' percent age dif- 
). r ^ >Jerencg in 3^tgigs giyeg h^. i "and J>anel; Z^fo^ the 'nine categories ^ 

' ^t^ (ii^ficu^ was 0 to 1, 1 . 

7 since : it^e\m^it5um dlff er'ettce^ would he between 0 percent (iOdj and 100 p : er- 
. <*cen* (iVOO), * : V ' ■ "• > \- - ■ , " . ' : 

/ j '^^'| f; ^:\ * '! 3 TABLg 30^- - • , / r ' ■. " • " ^ •. — ' . 

r;-.'"- ■ . 1'..' ? i . • ' • ■ -*'-■;''' '.. ' V.«'« ' ^-vv' « : 

;i " ''J-^tM^U • ; AVERAlp PERCE|j| OFaESSAYS ' 6LASSiFlM '*iN ^: 

i - ,* • ' EACH JUDGMENT' CATEGORY BY 'HALF-PANEL , l - • 



*■ i . .... „^j_ ■ .. , 



Panel 

'v. ■/••« '.. 


^ .Mean Score . 

, " .• - \- * 


..Perdekt..^ * .Percent 
"" Accjeptabie- ,. Unacceptable 


Percent '•• 
_ Bbrderline 


PaEiel 
^f^Panel 
To|al 
* 


■i ;; v^ 5.76 . ; • = 
2 ; ^ /."7.06 

Panels'. 6.91 - ^ ^ 

• a .• - . ■ • 

\ ; > ■ * ., - . ° • , , -f 

. > ■ . - » 


; 47.59 . -i, . .. 20.74 .• 
•. ' 4i.-67 V 28.70 
~ 44 ',63.. ' 24.72 

19 -;' . - : 5.. ■ 'i * . 


31.6? 
29, 63 
30.65 

7 ■ 

/ ' 






^TT^ • ■ rr 

. : ' s i 

" - v.* . • - 


/ 


• , -■«*.-. ' 

d «» !.\ 
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^ * tency wai measured here by comparing the number of panelists in each half ^ 
panel who placed essays in the three categories of > "acceptable, 11 "unaccepta- 
ble*" or "borderline. 11 . The distributions shown in Table 30 are . similar for 
the two half-panels. Members of Panel 2 placed approximately 8 percent more 
essays in the "unacceptable" category, while those in Panel 1 rated approx- 
: imately 6 percent more essays as "acceptable .": The average essay scores 
produced by the two half-panels ranged from 6-76 (Panel 1) to 7.06 (Panel 
2) , a difference of .30 points on a range of 0 to 12 . A T- test of this dif- 
ference found it to be statistically nonsignificant.. 

v Conclusions 

'The data presented in measuring Knowledge Estimation Panel consistency 
are interpreted as showing a high degree of agreement between half-panels 
about item difficulty, item relevance, and the score that could be expected 
of the minimally knowledgeable teacher candidate ,' ... The one except ipn to this • 
t general agreement occurred .in the relevance ratings for the. Listening Test. 
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CHAPTER V 
INFORMATION ABOUT 



The data presented to this point have addressed two fundamental- 



considerations in establishing a rational basis for the use of NTE Core 



scores in the certification process. —TO are: a) . the rela- 
tionship between the content 6f the tests and the content of the teacher 
training curricula >in Louisiana, and b) the estimated score level that 
would be achieved by a minimally knowledgeable teacher candidate. . 

; Information will now be presented concerning two other.; aspects of 
applying' NTE scores,, in the certification process:' -First, when the score 
standards derived from the Validation Team's judgments are adjusted to 
allow* fbir\ certification of the maximal number of qualified applicants, 
tlra ^isk of certifying unqualified applicants is^ inevitably affected. 
The level of risk (i.e., rejecting qualified applicants or accepting 
unqualified applicants) associated with each adjustment is included in 
this chapter. Second, the standard eventually adopted for certification 
will affect the supply of new teachers in Louisiana. For the present 
study , Educatipnal Testing Service has suppliecU-the...per f ormance distri- 
but ions of 4 Louisiana examinees .taking the Core Battery in November, 
1982, &nd March, 1983. To the extent that these examinees are typical 
of aii_ applicants for* initial certification in the State, these data ^an 
be used to estimate the probable effect that use of a given set of score 
standards will have on the supply of teachers applying for certifica- 
tion . Before addressing these two additional concerns, however > the 

p . - * ■ . .. - 

**■ ' ,*•.-.'• 

format of the revised NTE qualifying score will be discussed. 




;;. ,' \ V,, • . ^ . "V* ' , 100 p 

^RMAt dF REVISED MJE QU^tlFYiNS Se©6i . ^ 

When' the Ccfiamon Examinations Were used for teacher certification, 
purpos es; in Louis iaha f rom , 197 8 -1982 , the format of the qualifying score 
wa s the combined Area Examination and Weighted Common Examinations Total; 
" (WCET) scores . In other words, a candidate's Area and WCET scores were « 
. summed, and the total was compared to composite stanSarci, This model' 
had two salient features: compensation (a high WCET score could compen- . 
sate for ■ a low Area score, and vice versa), and an approximately equal * 
~ weighting of 'the Arei and WCET scores. Since the WCET was reported as a 
single score, the Area score carried greater weight than any single com- 
ponent subtest of the" Common Examinations . Within the WCET, the' follow- 
ing weights were assigned by Educational Testing Service to each of the 
component subtests : Professional ' Education , (4. 0) , Written English 
Expression (1.0) , Social Studies and Literature/Fine Arts (2. 5) , . and 
Science and Mathematics (2.5)/ 1 

As mentioned previously, the Gore Battery has a* format different 
from that of the Common Examinations;. The Core Battery consists of 
<• three two-hour modules , including a total of eight tests; the Commons 
was a single three and one-quarter hour examination comprised of six, 
... * tests . Educational Testing Service will report each of the tbifit Bore 
Battery module scores separately and will not weight* the modules differ- 
entially or combine them into a single. Core Battery composite score. 
Furthermore, the scaling of each module differs from the Area and Common 
Examinations scales: the- range of scaled scores for the Core Battery 
modules is 600-690, vdiile the Common and Area Examinations 1 score ranges 
were 250-990. .V 
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. ■ ' ." . • * ■■"■;>- 101 ' 

" > Th e NTE - Bfue l&bfibn S£ore i983i 4 66 

^dbtoipe^i a format .for the Coi-e Battery; and Area Examinations standards • 

x Although the Area Examinations were hot revised (and therefore were * not 

i ■ ' ' . . : - • - " ■ •\ ■ ... ' • j : ~" ■ ' -v. - ■ ■ r 'si ■ ■ . 

— ^ttire^sitbj-ect of the present validation study) , the ap p lication of Ar e a — : 

, sco'res in the ' certification process cbuld be affected by the differences; 

: r- — -, • - - - ■- - ■■■■■ 4-- • — ■ ■■ - ... ' - \ .. K 



between the Common Examinations and Core Battery previously ;h6t^.-;:.o£Ke.- 
Blue Ribbon Score Committee considered a number of format options for 
the new qualifying score(s)| including single 1 versus multiple, weighted 



versus unweighted, and compensating versus noncompensating score models.j^ : . 
A^er^ a lengthy discussion and the consideration of several alterna- 
tives, the committee voted in favor of a four-score, noncompensating. c 
format for the revised NTE standards. According to this model, each 
Gore Battery module and: the appropriate Area Examination would have an 
independent" minimum score that a teacher candidate must meet or surpass 
in order to be eligible for certification. The remainder of the discus- 
sion of qualifying scores will reflect this structural change f rom w the 
forper NTE composite score. v : ' 

i ;,/ r ^v::.;: • ^ 

RISK OF REJECTING .QUALIFIED CANDIDATES 

Each test s bore identified by the judges as the level that a mini- 
mally knowledgeable teacher candidate won \d : achieve is a theoretical 
true score. The score actually achieved' by any person taking the NTE 



falls within a range of scores on -either side of ^the examinee's true 

j. / 

score. One convenient way to interpret the score is to recall that, 

under certain assumptions, there are 68 chances in 100 that the exami- 



ERIC 



124 



See's observed Sbo.ire will be within bri^ standard error of measurement 1 

- ' * • - ' . . " "..« ■ • i' >:. .... * 

V of his or her true score* 95. -ctiarices in 1 6a that the observed score will 

be "within two standard errors bf measurement of his or het true score, 



aud— 99— chajices - : in— tOtT~that it will be within three standard errors o£; 

measurement of his or her true score. If the scores identified by the 

* _ . . . » * , 

' Validatipn Teato judges; are adopted as the standards for the Core Battery 
modules, some applicants with test scores below the Standards will be 
rejected even though their true scores, if known, would be above the 
standard. The reverse is also true. Thus, decisions in establishing the 
NTE standard should take into^ account the probability at different score 
levels of rejecting a truly .qualified applicant. 

Table 31 -presents the probabilities associated with the risk of 
rejecting qualified candidates with true scores greater than a specified 
value, and of accepting unqualified candidates with true scores 'of' less 

-than that 'value. The probabilities are given for several standards of 
acceptance^ expressed in multiples of the standard error bf measurement 

f " (SEM) . The particular multiples of the standard error of measurement in 

~ . ■ *■ ■ _ ■ ■ . - 

Table. 31 are merely illustrative; any multiple can. be used. The values 
in this ta^le are not specific to any particular test, but are general - 
ized probabilities; / V \ 

■ ' .,<-■' ' '. An example may help to illustrate^how^ahlei 31 can be used. Sup-* 
IpS^Q one were to consider adjusting the study estimate by subtracting 
one SEM from it. In this pase one would iofatel the, SEM value of -1 



-j 1 — • — «■ * ; 

The standard error of measurement is an estimate pf the amount of var- 
iation in a performance measure (e.g. 3 test score) attributable to meas'- ' 
• ureinent error. It is theoretically equal 'to the mean difference between 
examinees 1 observed score and true scores, \ j 

. ' . \ \ ' > 'i 



■ ,/' 125 
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; •■ table 3i' : v:^;^t: : ■' 

PROBABILITES OF INCORRECTLY REJECTING QUALIFIED' APPLICANTS - " •' .... 
AND INpRREeTLy ACCEPTING UNQUALIFIED APPLICANTS AT SIX 
LEVELS OF THE STANDARD ERROR OF MEASUREMENT (SEM) 



Probability of Rejecting Probability of Accepting 

Applicant Whose True Score . Applicant Whose True Score 

is the Specified Number of ; ±s the Specif ied Number of 
SEMs above Standard SEMs below Standard 





























SEM 


0.0 


0.5 


1.0 


1.5 


2.0 

- V 


2.5 


0 . 0 


0.5 


1.0 


1.5 


2.0 


2.5 


0.0 | 


.150 


.31 


.16 


.07 


.02 


\01 


.50 


.31 


.16 


.07 


.02 


'.01 


-0.5 | 


.'31 ' 


.16 


.07 


.02 


.01 


.001 


.69 


.50 


;31 


.16 


.07 


• 02 


-1.0 | 


.16 


• 07 


.02 


.01 


.001 


<.001 


.84 


• 69 ; 


.50 


.31 


.16 


.07 


-1.5 | 


r07 


.02 


.01" 


.001 


<.001 


*<.ooi 


.93 


,.84 


'.69 


.50 


.31 


.16 


-2.0 1 


.02 


.01 


.001 


<.001 


<.001 


<.001 




.93 


.84 


.69 


.,50 


.31 


-2.5 | 


.01 


.001 


<.00.1 


<.001 


<.001 


<.001 


.99 


.98 


.93 


.84 


.69 


.50 



(third row in the Table) v; Tfte left 'side of the Table indicates that 
there would then be 16 chances in 100 of failing to certify a candidate 
whose true score was equal to the specified score, seven chances in 100 
of failing to certify a candidate whose true score was one-half of aSEft. 
above tSe specified score, and so on. The right side of the Table indi- 
cates that there^wohld be 84 ^chances in 100 : of certifying a candidate 
whose true scofe was^ equal to the specified score, 69 chances in 100 of 
•'certifying ah -applicant whose true score was one -ha If of a/ SEM below the 
specified score, a3d sd on. 

Table 32 presents the mean score and standard deviation for each 
of th§ Core Battery modules for those persons who attempted the NTE Jh 
the f all of 1982 or the .sprjLng-of-1983 and -who identif ied themselves a.s 

_ ' _L _ ■_: _ _'_ ■_ _ _ _ _ ' _ ' _ s _ _ ." ' _ •_' ■ 

seniors at the time of testing. The data are also- limited to examinees 
who took the entire NTE (Three Core Battery modules /and an appropriate 



•: : - - ■ , — - ■ ' ^-r. ■„ ■ ; ' ' . . \ ' -^V V : ' • 104 

^rea v Ex^iiatibn^ 32 are the percent, of _ examinees ; 

aqhieving 'or (exceeding the miriimum; score proposed' by the Validation Teain 
judges on each module and the. standard error of measurement <SEM) for 
each module. This Table is limited to the percent -of examinees Meeting 
each individual standard. The .joint passing rates for ajll four stan- 
dards as a whole are presented in subsequent tables. 

TABLE 32 % : 

MEAN EXAMINEE SCORE AND PERCENT OF .EXAMINEES MEETING 
JUDGES' STANDARDS ON NTE CORE BATTERY MODULES 



J - 






Mean 






Percent 




Jucfges 1 




Examinee 






at or 




Minimum 




Score 


Standard 


above 




Score 


* SEM 


. (n=1013) 


Deviation 


Minimum 


Professional Knowledge 


652 


3.8 v 


656.5 


11 


.3 ■.:•'« 


67.-4' 


General Knowledge 


. 651 


3*5 


655.7 


12; 


:3 ; 


65.6 


Gommunicat io^Skil is 


."■ 652 - 


3:5 


658.9 


- 11. 


;*• 


74. 8 



NOTE: SEM derived from, fall, ^1982, form of Core Battery. 

♦ ' > . ' _ • V - ^_ ' • # v ■— 

Tab-Id 33 gives the percentages of Louisiana examinees s cor ingfe^bove the 

________ __ __ : . ■ 

oStudy estimate of the score standards for the three Corie Battery vmodules 

_______ ■ _ , _ ._ ■ »'■■■. \ ■_ ' ' *; & 

and each Area Examination. Tables 34 to 36 gffre the percentages of 
Louisiana examinees scoring above the study estimate of the score stan- 
dards for the three Core Battery modules minus one, two, and threes stan- c 
dard errors of measurement, respectively. ' 
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TABLE 33'. ; 

PERCENTAGE OF LOUISIANA EXAMINEES SCORING AT OR ABOVE 'STUDY 9 

ESTIMATE OF CORE BATTERY SCORE STANDARDS ° 
.H ■ . : BY AREA EXAMINATION 



v : 8 jJtB- ^ rea Number of 
.^^Examination ; Examinees 

• '^"WR^ > ; ! ' ■ j ^ /:J: . J ^ 


Standard 
-. for Area, 


1 

Estimate 
fqr PK 


Estimate 
7 far GK 


Estimate 
, for CS 


' _% 

Pass 


Agriculture * 


6 --; 


466' 


.652 


• 651 


652 : 


" 33.3 


■ Biology *& _; * : : . 

' General . Science " 




575 


'652 ' 


65 r 


652 


69.6 


' eBusiness ^ . 














* ^Education 




:591 


- 652 


651. 


652' 


: 45.0 


• Shemis try/Physic's/ 














• general Science 


/ 3" 


-"■530 V 


65 2 • . 


651 


652 


^100.0 


Early Childhood 












; . - ■ - ■ ■ 


* £. Education 


96 > 




. -652 


651. r ' 


652' ' 


43-8 


Education in 




■ .TV . 










Elementary- Schl 


: 511 * 


': 545 . 


652 


■651' '." ' 


-652 • 


54. Or 


Education of 












j * '_' * - 


nenrai Ketaraeq 


12 


541 


. 652 


651 


652 


33.3 


English Lang/- 














Diterature 




441 - 


( 652 


651 * . 


652 


83.7- 


French . 


'' . 2 ' 


517 '- .' 


. .', 65-2 . ' 


651 \ 


; : '652 


50.0 


German 


; 0 


» i+j o 


.DJ 2 








Home Economics 














.Education - ' 


29 ~ 


' - 509* 


652 


, 651 


, 652 


55.2 


tjatheraatics * 


° . 25 


617 7 ' 


652 


>651 ' 


* 652 : 


28'. 0 


fiefdia Specialist/ 












Library/ V- 


•'--••13 


563 


652 - 


'• : ;..65i.,. ; 


v 652 




Music - ' ' - . 














Education- *-\ 


- ~.-46 


533 V 


552 ; v 


'. 651 


652 " 


58. 7 - 


•-■^^•••^Pl^'sicai • 


s 












s Education : 7 


■ 113- - 


- 545- '•: 


- 652 * 


; v .651 


652"' 


. 36 .3 


; ; . y'; :* : Spc ) ial Studies ..'*".' - 


- 46 


563 


65*2 


' 651' 


" ' 652 


52.2 




.4 


538 


6S2/: ' 


651 


. 652 • 


75.6 : 


'Speech ; Commtiiiic J; 














^ - Theatre (013^ - 




: 519 , 


65^1' 

.. ..*?.?&• 


. 651 


'■■ .532 . f 





Total Across Areas 1013 ' . — "r- ' 1 ' ;:652W^s- 651 .652 ' ' -si-S 



PK=Prof essibnal ^o^iedge;; GK^General ^owle^^^CS^dti^^ 
NOTE : See text fbir- exf>l^iiti6n : bf Area Examiriatlbin Standards '., > ? : 



TABLE 34 - 

' " • :.. . - ... ■ - • •'«'. 

PERCENTAGE OF LOUISIANA EXAMINEES SCORING AT OR ABOVE STUDY- ■ 7 ' : - ^ 
• •: ESTIMATE 'OF CORE BATTERY SCORE STANDARDS . MINUS ONE • . 

. :• : ■ ■ ■ . BY AREA EXAMINATION • • ■ ■ ■ 



Area w Number of 


Standard 


' Estimate • 


Estimate^ 


* 

Estimat e * 


• ■:; -': 
'% 


Examination Examinees 


for Area 


for Pk 

— — — > . .■ 


' for GK 




Pass - 


Agriculture 


6 


466 


649 •". 


/ 648 


649 V 


50,0 


_Biolb|y V 














r ; General Science 


23 


575 




648."';"' 


£49 • 


73.9 


Business 










•' '•• ./ " r ''' ' ■ 




Education ' 


40 


591 


'649 


648 ; 


^ 649 ... ' 


55.0 


Ghemist ry / Phys ics / 







■ - - - . 






■ ii'L. i - ..." 


General Science 


■?.•;;'■■ 3 


> 530 _ 


649 


648 ; y 


7 649 ' 


.4.00.0 


Eirly Childhood 








/ 






± Education J 


96 


' 506 


649 


648 ' 


649 


5o.o 


Education v iij. 














Elementary Scfil 


sii 


545 


649 


, 648 


;" 649 


62.4 


Education of 








■ "•' '■>;• ' 


> 


■ . ■ : . > 


Mental Retarded , 


; 12 


541 


649 


648 


649' ; . 


,50.0 


English Lang/ 




* _ _ 


- - — „ 3 








_ Literature * 


i. - 59 


441 •* 


649 


-' .648 ' 


649 


,91. 8 


French 


2 




649 


< 648 


649 , 


50.0 


German ; ' *' > 


0 


"496 


649 . 


648 • 


: 649 




Home Economics 














Education; . - 


29 






0*fO. 




05 .3 


Mathematics 


25 


617 . 


' . 649 . 


'648" 


649 


• ' k," 

.32:0 


Media Specialist/ 














• Library/ A- V- 


' 0 


563 


659 


; 648 


649 




Music f* . • 










A 




-Education 


46 


533 . 


649 


'. .648 


649 ••' 


69.6 


Physical 














: Educat ion 


113 


-lis • 


649 


648 


649 


47.8 


Sbciii Studies ' 




563 


. 649 


648 


649 


54.3" 


Spanish • . . t :. 


4 


$38 


649 


648 , 


649 


.75 . 0 


Speech Commuivic 














•& Theatre (OLD) 


8 


519 


649 


648 m 


. 649 


75.0 


Total Across Areas . 


1013 




649 


648 v 


649 . 


60' . 3 , 

















PK=P r o f es s iona 1 Knowledge; GK=General Knowledge ; CS=c6mmunicat ion Skills 
NOTE: See text for explanation of Area Examination Standards. 



. . TABLE 35 



PERCENTAGE OF LOUISIANA EXAMINEES SCORING AT OR ABOVE STUDY 



' - ESTIMATE OF CORE 'BATTERY SCORE STANDARDS MINUS 


TWO SEMS 








BY AREA EXAMINATION 


; 




— — . 


Area Number of . 


Standard 


Estimate 




Estimate 


Of 


Examination Examinees 


for -Area 


for PK / 


tor bK 


for CS 


Pass 


Agriculture 


6 


466 


645 


644 


645 


83,3 


Biology & 

General Science 


23 


575 * 


645 


AAA 


645 


78.3 


Business 








- - - * 


f 




Education 


40 


591 


645 




645/ < 


57.5 < 


Chemis try/ Phys ics/ 










■ • 




General" Science 


3 


530 


645. 


644 


645 


100.0 


Early Chi ldhooji, 














r Education 


96 


50,6- 


645 


644 


645 


63:5 


Education in 


• 












Elementary Schi. 


; 5ii 


545 


" 645 . 




645 


68.7 


. Education of 






_ _■ _" 








s Mental Retarded 


12 




541 


645 


644 


,645 


50.0 


English Lang/ 














Literature 


49 


441 


'645 


644 


645 


91.8- 


French 


'2 


517 


645 


644 


645 


50.0 


German 


0 


496 


645 


644 


645 


** — 


; Home Economics 














^Education 


29 


509 . 


645 '. 


644 


645 


86.2 


Mathematics 


25 


617 


645 


644 


645 ' 


32.0 


Media Specialist/ 














, Library/A-V 


0 


563 


645 • 


644 


645 


-- 


Music 














«. - 
_ Education 


46 


533 


645 • 


644 


645 


80.4 


Physical 

Education " 


113 


545 : 


- V 645 


644 


645 


66.4 / 


, Social Stttdies 


46 


. 563 


645 - 


644 


645 


58. 7 ( 


'Spanish 


4 


■ 538 


645 


644 


; 645 


75. Q . 


Speech Communic 














• & Theatre (OLD) 


8 


519*' 


* 645. 


644 


; 645 * 


75 . 0 


'Total Across Areas 


1013 




645 


644 


645 


68.5 



PK=Prof essional Knowledge; GK=General. Knowledge; GS=Commuhication Skills 
NOTE : See text for explanation of Area Examination; Standards. * v 
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•• \ . . Able 36 . 

" ' • ' ; ' : • • . ' 1 . ... \ 

i- . • PERCENTAGE OF LOUISIANA EXAMINEES SCORING AT OR ABOVE STUDY 

- ESTIMATE OF CORE BATTERY SCORE STANDARDS MINUS THREE SEMS 

BY AREA EXAMINATION 

- ; ._ \ _ - / ;■ . . . 



Area, Number of Standard Estimate Estimate Estimate % 



Exarainat/ion . Examinees 


for Area 


for PK 


for GK 


for CS 


Pass 


Agriculture ~* ■ 


6 


466 


641 - 


641 


'642 


83.3 


Biology & \_ 














, General Science • 


23 


575 


641 


'641* 


• 642 


■ 78.3 


Business •; 














Education 


40 


591 


641 


641 


642 


60.0 


Chemistry/ Physics/ 














General Science 


3 


530 


641 


641 


642 


100\0 


Early Childhood 












\ 


_ Education 


96 


" 505 


641 


641 


642 


7 2tf 


Education in 














Elementary Schl 


511 


545 


641 \ 


641 


642 


73.2 


Education of 














Mental Retarded 


12 


541 


641 


641 


642 


58.3 


English Lang/ 














J_ Literature 


• 49 


441 


641 - 


641 


642 ' 


\ 93.9 


French ; 


2 


517 


641 


641 


642 


50.0 


German 


> 


496 


641 . 


641 


642 




Home Economics 












Education 


29 


509 


641 


641 


642 


93.1 


Mathematics 


25 • 


617 


641 


641 


642 


32.0 


Media Specialist/ 














. Library/A-V 




563 


641 > • 


641 


642 




Music 














Education 


46 


533 


641 - 


641 ' 


642 


82.6. 


Physical « 














Education - 


113 


545 ... 


641 


641 


642" 


75.2 


Social Studies 


46. 


563 " 


641 


641 


642 


58.7 


Spanish 


4 


538 


641 


641 


642 


"■75.0 


Speech Communic 














& Theatre (OLD) 


8 


519 


641 


• 641 


642 


75.0 



Total Across Areas 1013 ---- 641 641 642 73.2 



vPK=Prbfe 9 ssian^ GK=General Knowledge; CS=Communication Skills 

NOTE: See text for (explanation of Area Examination Standards. 
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The standards for the Area Examinations in all four tables were derived > 

from the 1978 validation study estimates. These estimates were not used 
,tp establish independent standards for the Area Examinations in the 1978 
.composite score model. In order to allow for the effect of t the tran- 
* sit ion from the former, composite (conpensating) model to the" present 
noncomposite model, a s statistiqal adjustment of three SEMs from khe 1978 
study estimates was necessary.- These Area. Examination standards appear 
as constants in Tables 33 through 36 and the remainder of this report. 

Only those Area Examinations that are required for certification 
ia. Louisiana and. that were judged to be valid in the 1978 study aire 
included in the analyses presented in this chapter. Candidates -seeking 
teaqher certification in the areas of Speech" "Pathology, Art Education, 
and Industrial Arts Education will be required to meet the standards 
established on the three Core Battery modules but will hot be required 
to meet an Area Examination standard. Candidates graduating from the 
Generic Special Education programs in .Louisiana will also be requiredXto 
pass only the three ^re Battery modules. ' v ';-:{' .' ^] 

Educational Testing Service has also revised the Speech" Communica- 
tions and Theatre test, producing' a new Speech Communi cat ions Area Exam- 
iiiatioh to replace it . Candidates seeking certification in ,'this area 
and taking the new Speech Communications Area Examination will be 
required to meet-bn^y the Core Battery module standards- until the new 
Speech Communications Area Examination has been studied for validity' and 
a standard set for the test. Candidates seeking certification in, Speech 
Communications after the new standard has been set will be required to 
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meet the Area Examination and Core Battery module standards<& A valida- 
tion study of the new Speech ^ ebmmunicSt Ions ^Are*a. Examination is sched- 
~ . uled for the fall of 1983. ~~ ' ' " ~ ~ ' 

It should also be noted bh Tables 33 through 36 that no Louisiana 
^ £ examinees attempted the German or the Media Specialist-Library Audio and 
Visual Services Area Examinations in the fall df 1982, or the spring of 
1983 , .As a ^result Tables 33 through 36 ! db ^it. report estimates of the 
percent of examinees expected to meet the proposed Core Battery, module 
standards .in th^se areas. ; 

; " SUPPLY OF NEW TEACHERS IN LOUISIANA 

^ ^Closely related to the consideration of the risk of rejecting 

qualified candidate^ is the assessment of the effect of proposed qualif- 
ying score Requirements on the supply of new teachers in each of the 
specialty fields. If there is fi a substantial shortage of teachers at i 
particular time J a higher risk of certifying unqualified candidates, may 
be considered acceptable. Information oh teacher. supply and demand may 

----- --- .. v - - x - ... . . ■ a 

be used ~in conjunction with tne validation data reported above to esti- 
m mate the effect of any set -tff score standards on the probable supply of 
.. candidates and, indirectly, on the capability of meeting demand. 

Tables 33 .through 36 give selected ; statistics based on Louisiana 
seniors who took the ME /Gore Battery Jn the fail of 1982 or the spring 
•: . of 083 1 If an, examinee had taken any of the Core Battery modules or 

• • W d ■ ' _ * ■ 

the. Area Examination more than once, only the first scores earned were 
included. This: procedure is consistent with the ♦ 1*978 NTE validation 



study in Louisiana'. There was brie exception. . If , an.* examinee who Kkdt 
previous ly taken an Area Exam ination attempted it again in April, 1983 3 
the more recent Area Examination Score was used. * ' 

, The values of the means of. the estimated scores 'cannot be compared 
actoss teaching areas, since the scores on different Area Examinations 
are not on comparable scales: The percentages of examinees whose scores 
exceed the ^estimated means can be compared, although in' interpreting the 
comparisons it is essential to recognize that / different fields may 

attract students with dif ferent ^>leVe Is of talent ; - Moreover 3 . the : numbers 

".*■**,"■ ■ ■ ■ *«* 

of examine es who take some of the Area Examinations are relatively 
small s so that the results for a different period could vary appreciably 
from those observed . 

- As shown in Table 33, the application, without adjustment of the 
NTE score levels suggested by this' study, would result in the certifica- 
tion of different numbers and percentages of candidates in different 
teaching fields v On the average, only 51.8 percent of the 1013 exami- 
nees whose scores were tabulated would be certified accordingly. His- 
topically* validation study estimates of ^performance standards often 

\ - }_ _______ __ ' . _ ■ ' . ' : . . c 

yield actual passing rates that are unrealistically low, given the 
demand for new teachers in many states. In keeping with the report of 
the 1978 study and its implementation by the Blue Ribbon Score Committee 
and Superintendent of Education at that time^ only deviations from the 
study estimate that yield higher pars sing rates (i.e., by subtracting 
standard errors of measurement) are included in the present report. The 
effect on passing rates bf\ standards that .are more stringent than the 



f- :;■ in:. 
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study estimates (i.e.., by adding standard errors rff measurement}/; Mil be 
available to the Blue Ribbon Score Committee -and ttie 



Education , if and when] requested . 

•' . • .«■"., " .» ■ ■' . 

" /.'... . .' . '\V '' ' & \- . _ ■ 

; Finally, it should be restated that the projected passing ratgs 

: presented in this ; chapter are based upon the 1013^ Louisiana seniors who 
were examined on the. Core Battery modules and an appropriate Area. Ekami- : 
j nation. This method of analysis is^ consistent- with the 1973 Validation 
study. However; the total number of individuals seeking teacher certif- 
ication in a given year may exceed the 1013. included in this study's 
population. In comparing the Staters need for teachers with- the percent * 
of candidates in^the. validation population who would meet a given set of > 
standards^ it is important to recognize that :tfie total number of certi- 
fiable teacher candidates will almost certainly exceed the number of 
college senior 'teacher candidates who would pass each set of standards . 
,. Thus, while the passing rates -discussed in Tables 33 through 36. apply to 
graduating senior teacher candidates,* the actual number of certifiable 
teachers will include persons in addition to college seniors "and will t 
very probably be somewhat higher than the numbers, shown' here. 



8 




■ % 



ERIC 



CHAPTER VI 
SUMMARY OF THE STUDY 



This' chapter is a .summary of the preceding text . It is' intended 
as y a brief synopsis of the findings and an overview of the entire/study. 
Earlier chapters should be reviewed for complete documentation and more 
detailed explanation of the findings summarized below. * .. " » ■ 



STUDY DESIGN 



The purpose # of this study was to evaluate the validity of the NTp- 

Core ' Battery for use in certifying teachers in Louisiana. The design 

... . .. . .. .." •»_ , * •> • . ' ' 

was modelled on the one developed by Educational Testing Service and 

used in Louisiana's 1978 validation study of the ent ire~OTTir; In; the - 

current' design, panels of faculty .from Louisiana co 1 f eges and vunivers f - 

' .. \ ■ ., . :.- ' • .... - * ■/./#. ., ■ V-..: 

ties with teacher education programs reviewed the eight tests 'comprising 

' .... if- ' ' ■ ' :.. ■ 

the NTE Core Battery. They provided judgments about the f o 1 lowing* nia j o r 
-aspects ^bf the Core Battery as an instrument for teacher cej*tificatiotf; ; : 

_ i_ ^ Soht ent Review . ; Is the emphasis given to topics .' 
within each test the same as the emphasis these t^ics ? ; 
receive in Louisiana teacher education curricula?* ,;'t^ld ' ' 
graduates from. Louisiana teacher education programs hake 



the opportunity to learn the content included in . eac^-test^ / 

item? '•" A - " ; : ; ' , ' '^^^'^^ " 

Knowledge Estimation . Would the minimally khbwled^er^ 
able teacher candidate find each test item easy, mod^ra^ely 
difficult, or hard? Is the knowledge measured in e0k. test : 
item essential, important , or not very important? v;?^at pro- ■ 
portion of items in each of the nine resulting .categories . of.. 
-difficulty and relevance would the minimally : knowledge^ 
' teacher candidate answer correctly? ; ; I "■ r ! 



* - ' • . 1 /' 114 

TOfe ' tasl^ of the Content Review Fane L. was to provide information 
Regarding the match between the content of the tests and teacher edtica> 
* «v tibri curricula. This ensured that the estimated scores would include 
^bnly those items that were appropriate measures of. Louisiana teacher 
education curricula; that is,. those that reflect content the teacher 
candidates would have had" the opportunity to learn. The judgments of 
this Panel were combined with those of the Knowledge Estimation Panel to 
produce estimates of the score that could be expected* of the minimally 
knowledgeable teacher candidate on- each Core Battery test. The result- 
ing scores were then 'combined to produce estimated scores for the three 
modules of the Core Battery: Professional Knowledge, General Knowledge, 
and Communication Skills. The Blue Ribbon Score Committee also met dur- 
iiig the t-ime the study was under way and recommended that the required 
NTE\ score take the form of four .minimum scores (one for each of the 
three Core Battery modules plus the appropriate Area Examination) in 
determining a teacher certification standard. 



STUDY PARTICIPANTS a 

The usefulness of the study design rested largely upon the degree 
■ which the ;£&culty who served on the Content Review and Knowledge" 
Estimation Panels x^as representative of Louisiana teacher education pro- 
grams. As Chapter II illustrates, tllis objective was achieved. Twenty- 
one of the 22 colleges and universities with such programs were "included 
in the Panels, and\ the' number of faculty participating from each insti- 
tutibn matched the institution's size in the number of graduating stu- 



v : - ''. us 

dents. The Panels also reflected an equitable' representation of , private 
institutions and those with .predominant ly black student bodies. For all 
Core Battery tests the number of faculty participating in both the Con- 
tent Review 'and Knowledge Estimation Panels met or exceeded the number 
required for reliable results. 



-_«_ 7 %, V x4 —— v- - . 



CONTENT REVIEW FINDINGS 

The 'results of the Cdntent Review Panel jud^ehts ^are presented 
/ "..» here for the three. Core, Battery modules arid for each of the eight compo- 
nent tests. Thes,e summaries include the content appropriateness of te£t° 

^ . . .• . t ; _ _ _ • _' _ . . _ ■ 0 ■ 

items, the comprehensiVei^ess of the tests, relative emphasis given [top- 
.^ c ^ ^ ^ e tests and in Louisiana teacher ' educatioii curricula, and the 
overall similarity between the tests and these cuij^cula. Chapter III 
provides a complete discussion of the procedures and data upon which, 
these summaries' are based. * _ ^ 



Prof essional 



This module consisted of onjp. test , Professional Knowledge. It was 
* - ~' \ ■■ ' * ■ • <^ 

judged to be very closely related to the Louisiana teacher education 

curricula. Ninety-three percent of the items were judged appropriate 1 , 

and therms were no topics- noted by two or more judges as omitted froni the 

test. The enrphasis given topics within the test was acceptable. When 

there was a .difference in emphases, topics tended to ; b,e. overrepresented 

on the tefst, in comparison with the emphasis they received in teacher 
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education programs. rEigHty- eight percent of the judges ffelt that the 

t. « : - / I ,...,...:.3^, i;;;< ■ . j " _ . : ' 

test closely paralleled, or di'd^'npt;. significantly from Lbuisiah^ 

teacher education curricula. V / ? 



General Knowledge Module 

The Senerai Knowledge Module included four tests: Mathematics , 
Science, Social Studies, and Literature/Fine Arts. iThe module was /eval- 
uated as closely related to Louisiana teacher education curricula/ Each 
component test is discussed below. 



Mathematics Test * j 

Ninety-eight percent of the items were judged .content appropriate. 

Three topics (probability, statistics, and algebraic problems) were 

• * • ■ . / 

cited as omitted. The relative emphasis was acceptable, and where there 
were differences between the test and curricula, topics wire oyerrepre- 
sented on the test. Eighty percent of the judges rated, the content of 
the test as a close match with* the teacher education curricula, > 

Science Test .- * * j : % 

Ninety-seven percent of the items were "judged content appropriate/ 
No topics were noted by. two or more panelists- as* omitted. The relative 
emphasis was acceptable, and ^wfiere differences were noted, s topics were; 
"generally overrepresented on the test. The test was perceived as match- 
ing th'e content of Louisiana teacher education program by 90 percent of 



the panelists . 
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Social Studies Test 

In this test 100 percent^ of the items were judged to be content 

' - - y . . .. * • , • ' ■ ■' 

appropriate. No topic was listed , by two or- more panelists as omitted 
from the test. ^The value for relative emphasis was .48, less than the 
• 50 level set for exclusion from additional review on this criterion. 

Again, when panelists felt that the emphasis given topics on the test 

« * 

differed from that given topics in teacher education curricula/' they 

... . • . . . ' ■ _ ' ■ V ' _.. • - . * 

judged the topics to be overrepresehted on the test. However, some 82 

percent of the panelists evaluated the test as Sioseiy paralleling, "'or 

not differing significantly from, "Louisiana teacher education Myograms . 



Literature/Fine Arts Test 

Eighty-four percent of the items were evaluated as content appro - 
priate. Five of the 10 items judged inappropriate dealt with a single 
topic, "relating works of art. fo one another aiid their social-historical - 
* context. 11 One topic ,v ; recognition of the names and works ;6f famous peo- 
ple;- was noted by two judges as omitted from: the test. The relative' 

- : ... _ . . .... " ■ _:: . . v" . - . _ , ■ 

^emphasis was .43, - less than, the .50 set as ; an acceptable level . When 
there was a difference between the emphasis topics received bri the test 
d in Louisiana teacher education, curricula, topics : Were generally 



overrepresented on the test. Sixty-seven percent of the .panelists 
judged that the content of ^he ' test matched that of teacher education 
curricula closely or with no significant differences . '"^ . ^ : 
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Communication Skills Module x V 1 

• . " 7 J * y • " 

This module % was^udged to be closely .related to Louisiana tocher 

education programs . "Sit. included thr-ee *te£ts: Reading, Listening, and 

Writing. The .Waiting Test, was/ in\turii, composed of .an* objective, sub- 

test and an essay ^exercise. 'The essay 4 component was* hot considered by 

the Content* Review Panel because it did not include disbrete items that 

cpuld fie evaluated for their appropriateness or for their match with the 

emphasis topics received in teacher education curricula. < 

Reading Test -V :"' " . * . ; ■ " 0 

All of the items (100%) were evaluated as content appropriate, and 
ho topic was.* felt to' be omitted. The relative emphasis that topics 
received "bn^ the test was judged to be acceptably 'close to the emphasis 

these Jfcpici received in teacher education curricula. Differences *in 

■>\ 7 . ' . ;. . - 

- ... if. * 

emphasis reflected an overrepresentatipn of topics on the test. Ninety- 
two percent of the panelists judged that the test closely paralleled; or 
did not differ significantly from,, the content qf Louisiana teacher edu- 
cation programs . . 

'^77: 7 '7. 4=7.7" t " : V7v7 ; ' : " = 

Listening Test 4 i 

Ninety-eight percent^ of the" items were'' rated^ as,* content aipprppri* 
ate. Two topics were cited as omitted from the test: analysis aid 
synthesis of oral communication, and stimulus -response questions for 
communication. The relative emphasis was .36, indicating tti&t less ..than 



,5 0 percent ' of the panelists judged that - % topics were given the ; same 

emphasis on the test that they 'receive^*in teacher education programs; 

: ' '• * • ' • , .... Wr " " • 

.'. " • " • ;. ***** ; ■ *.-....• . '■' 

Differences in emphasis\weire generally cases in which topics were bver- 

represented 'on the test. Forty percent of the panelists rated the test 

• '•;« ' ' . . »\ . -V' - ' ' ■ .• . ' . * 

as closely paralleling,* pr not differing significantly, from, Louisiana 

■« * v. ■ v*""" 5' '' : - : ; : * _ . 

teacher education programs. 4 ^ 



WHtirfg test " ' .> - 

'As 'not;ed earlier, Content Review was limited to* the objective sec- 
tion of . this % testT * Ninety-six percent of the items were judged to be 
content appropriate. One, topic was noted by two judges as omitted: tlie 
spelling of frequently confused words (tor- too, threw-through),. The rel- 
ative emphasis value was an acceptable .50. i When the emphasis given 

__ ____ ■ _■_ ' _L ■ il ' jiC :C /• *m_ • 

topics on the test differed from that given in the ^curricula, jpopics 

\ __ :* v * ' ■ '__ 

were overrepresented on the test. In judging overall similarity,- > 64 
percent of the panelists evaluated the~ test as matching the. content of 
Louisiana teacher education programs . • , ■ 



KNOWLEDGE .ESTIMATION i 

.The Knowledge "Estimation Panel members evaluated the difficulty 
and inipbrtaice of each ' item in the eight &ore Battery Te* ' >.Tfiey^€fiei 
estimated the proportion bf items that .the minimally knowledgeable ; 
teacher candidate would answer, correctly. The method used for evaluat- 
ing %he essay exercise of the Writing Test was somewhat different - 9 • since 
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v • - ■ * • ' • * ' ■ * 

the essay was not compos eel" of individual' items.. For -this exercise, 

^ }_ * _r\ vi _ * 

panelists reviewed a number ^of essays that^had been written by Louisiana 

- * - ■ ' *. - - - . C) . . _ * . 

teacher candidates attempting the. NTE Core Battery, and classified- each 

-as acceptable, unacceptable, or of borderline acceptability. - \ 

The proportion 6f items at each level bf difficulty and importance 



that the minimally 'knowledgeable teacher candidate could be expected to 
answer correctly was then combined, for each test, using only those 
items judged to be appropriate by the Content Review? Panel. This proce- 
dure ensured that Louisiana Teacher candidates would not be penalized 
for content they had hot had the opportunity to learn. However, rteacher 

candidates could still earn points toward their total score on a Core 

, > '.' . . " <>\ 

Battery module by correctly answering those items that had been rated as 
content inappropriate by the Content Review Panel, 

j£ The resulting a scores that could be ^expected °of ithe minimally 
knowledgeable teacher candidate are shown for each .dfcditffe on "tteb^e 25 in 



Chapter IV. Chapter V discusses the impact of these jjcores Louisia- 
na 1 s supply: qf teacher candidates. Chapter, V also, shows the percent .of 
teacher candidates who could be expected to attain each set of minimum 
scores as well as variations from the scores expressed its standard 

* — - - - ' » 

errors of measurement. These estimates were obtained from the perform- 
ance of Louisiana teacher c andida t ejs*V ^n th% fall, 1982, ;^and :Sprijig v > 
1983, administrations, of; the NTE Core BaftteryV * 
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SUMMARY « 

' - the Content , Review section jof this study 'found the NTE - Core Bat- 

o. . 1 . _ i • . .. V .;' * ' .* _ * * __' _ .. • _ 1 . 

tery modules- in .the judgment of those, Louisiana college and university 
faculty members who evaluated their component tests, to be valid meas- 

ures^ of teacher education curricula in the State. The Knowledge Estima- . 

- • " - ? 

tidii results present scores for each module that in. the judges 1 estima- 



tion cbul;d be expected of a minimally knowledgeable /teacher candidate. 
Information is also ' provided about the potential effect of possible 



Lifying scores on the supply of new teachers for Louisiana. All of 
this information is presented to the Superintencffent of Education and the 

' •. . •• ' . . / _ 

Blue Ribbon .Score Committee to -assist them in their deliberations about: 
appropriate performance standards on the NTE Core Battery for certifica- 
tion • of beginning teachers in the State. 
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OVERVIEW ^TASKS TO ^PERFORMED BY MEMBERS 
4 )M^F: GONfENT REVIEW PANELS 



The study in which . you have been, asked ; to participate is -being conducted by the 



j^ouisiana^St^ie^ „ 
evaluate the content of the National Teacher Examinations (NTE) ■ in r ela t ion to 
Louisiana teacher education programs and; to estimate the test "performance, of 
minimally _ knowledgeable / candidates for certification as- teachers in t hi public 

schools in Louisiana ^ "■' * ^ 

■ ■ ■ * . - ^ » " .# ri* ■ ...... 

A Content Review Panel has been established for each section of the" J^TE Core 
Battery Tests. : You . have been selected to serve on the Content Review Panel 
for— .. - . ■ •' .. i. , v ■ ■ '. ■ -. * ' _ - ^ • 



As a membfer of the Content Review Panel, you will be asked, to perform two tasks:' 

$ : 1 v To ea^ine ' content that, is ifpllowed in 

developing the test and to ascertain whether the curriculum areas covered by the 
test "are congruent with the curriculum areas covered by the program at your 
institution :br : - btheir Louisiana Institutions with which you are familiar/* 

. 2. To examine individual test questions and to ^ judge: whether the content 
of each_. question would have bejeii included ^ progtam(s)^ 
followed by the students who take the test for certification. ^ 

^_j^* cate 4 i° the NTfi Design St^aaryi a number of faculty ''members-.. .^iptlT Re- 
assembled in order to make their judgments. The Judgments, however, %ill^fcfc^ 
niade individually and independently; members of the same panel will not confer r 
as a group, nor will any member be informed of the judgments made by any other 
individual member. : : The judgments of all members of a panel; will be combined 
statistically by the fcSD E to arrive at ^ sugary j^^ent for: the, panel about 
each question. The ^sjammary judgments for the. quesraons also will be. combined, 
and the final results; will be published in a report describing the study and - its 
findings or conclusion^ . 



The information in this mailing is intended to help you to prepare fox your 
tasks." If in studying the materials you find that you have ^questions about the 
tasks* be sure that they are answered during the initial orientation session it 
the central meeting site. j * . vJ'.i/' 

The Test Content. , Description identif ies the ma j or groups of topics that are 
covered by the test; and indicates the relative emphasis that is given to each. 
a Y6u will be asked to evaluate the overall congTOenae^etween the content of the 
teacher education curriculum and the content of the test, and to record your 
evaluation on the Test Content Review Form. In addition, you will be given a 
set of test questions and asked to make judgments about them and to record "9pur 
judgments on the Question Review Form, Before you go to the central meeting 
Site, please thitik carefully about the Test Content Description and the 
curriculum at your institution » You may wapt to make preliminary notes while you 
have access to various sources of information on your own campus and to bring 
them with you to the meeting site,* where you will be asked tp complete the 
Review Forms. ■ .' 



You have been asked to participate in tKii study because you are_ familiar with 
the_ curriculum, at your iiistitutipit in the field (s) covered , by the^eicjaMhatibn 
with whi^h you- will be wrkiiig; '-Before attending the panel ^sess£6n^ you. Say 
want tbjdraw upon lbcal_ sources bf infbnnation regarding the curricuium^at yoyr 
institution, such as. your college catalogs, specialists in curriculum planning, 
or; other sources available to you. " You "may also find it helpful to talk with 
^^^agies^^ 



to teach. 

Your^^^ this istudy and your qualifications to participate are;an 

• -imiportant part of the 'study jnethddology,; In order that the final report of the 
\itjUdy^3 ^findings and conclusions he as. informative as possible to other 4 who i tnay 
^yish to/use- it, «:we ^111 ask> for ybur^pennission to identify you in the final 

report^ £v %- - .. r •? 



cftt : n f - ma teri a ls -wit'h ^you .- 



t^^e^ij^i^appreciate your willingness to participate in this important study./ 



0 Rb££rt*Jv GaiVue 



NTE Validation Task Force 
Louisiana State Department of Education 
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• , " INSTRUCTIONS 
j : j ' QUESTION REVIEW FORM 



Tour task ia to make judgments about whether or not .the content of individual 
test questions ^ that ar? part of Louisiana 

' teacher education programs le&iing to certification, , > ^ 

■r In making your judgment about each question, confer whether or not the Content 
pf the question would hive been covered in any of the courses normally taken by 
students in 'teacher education programs. In some cases a course that is prereq- 
uisite to entering a^ in college by 
aome atudents biit ^ high school by others . When such a course is the brie in « 
V which the ~cort tent ofTY "particular question ir^ all~stuaent s 7who~ w5gia~ 
v liave taken the course, whe titer in , college or. in high school, shouTj h* con- 
sidered to have had the opportunity to learn the coritenti You are not to judge* 
whether the students would, 15 facj^ have^ learned the answer; you are only to 
judge whether they would have had an opportunity to,lea*n the answer. 

>As you read each tept question and its underlined ^ «iswr ^ If^e whether at Te^ast 
90 per cent of the students in the group with which yoUy»e; concerned would have 
had an opportunity^ to learn the answer. If you think tliat they would have, 
circle "Yes" on. the QueStion Review Form; if you think that they would' not have; 
- circle "No". Before you circle jrour .answer, please make sure that the number 
that identifies the question on the answer form is the lame as the number that 
identifies, the question in the question set. **\ 

</If_ you feel that your ^ you with basis whatsoever for 

making a judgment about one of the questions , you may ■ tircle^DNK" (for "Do Not 
Know") . The DNK category is to be used simply because ydu have difficulty 
in deciding whether to answer "^Yes" or "No"; you are to ^ke^ decision even if 
it is a difficult one. The DNK category is to be^uged^biily when 1 you have no 
basis for making any judgment. >\ v^' « ^ 

In making your judgments you are not< to be concerned about how many questions 
you are assigning to the "Yes" category or to the "No" category. It is your 
responsibility to apply your best judgment in evaluating each question indi- 
vidually. ■ v . ■; * 

A| ger you h^vfe, finished making ^your judgments about the questions in the entire 
question set, please look over tte and your decisions about them to 

make sure that you are | satisfied with the consistency of your judgments. A 



* * * 



~y . ■ ■ ■ a. s _, y ~ _ ■■ 

***** operators who 1,111 be " keypunching the forms will appreciate yoyr using a 

Ntffifcer 2 pencil and erasing carefully when you change your mind so that your 
% final judgment about each question will be clearly indicated. * ; ' 

As ■ you ; turn to each new test item, please be sure that the number of the 
question on the page corresponds to the number on the Question Review Form. When 
you have made judgments about all questions in the .question set, check^at the 
wn§^o|!_.^e;b8t question for which you have recorded a judgment on the form, 
© corresponds to the number of the last question in the question set* r 



as ■ 



■,../> QUESTION REVIEW FORM . ? '% '% 
, professional Knowledge: SECTION 1 0 ■ 

. : - __ ._ _ * / « . A 

Judge's Name t - r ^ , v» : 1 Form 

* . , * ■ \ • ' 



Question Is Content , Question . - Is Content 'A/. Question. Is Content 
Number .. Taught? " Number.. 'Taught? Number Taught? 



ii Yes No DNK 


13. Yes No DNK - 


25. Yes No DNK 


2. Yes No DNK 


14. Yes' No DNK 


26. Yes No DNK 


' 3. ' Yes . No DNK - 


15. : Yes^ No > DNK • • 


- 27. • Yes ; No DNK 


, 4. ' No DNK 


16.^ Yes No DNK 


28. Yes No DNK 


? i 5 V Yes No DNK 

_ . - ^ - . •? ■■ .-■ ■ 


17*. Yes No DNK 


29. Yes No DNK 


6. * ; Yes No DNK 


18. / Yes No DNK 


30. Yes v No D'NK 


• y _ . . ... . .• ; .r ■ ' ;-r 

. 7. ' Yes No DNK - 


. . ••" • ' ¥ .'-v : ' '".V ••••• 

19. ■ Yes No;. DNK / 


31. ; ' Yes NoN^C^ ::, ■ 


' : * Yes : f«o DNK - , ' : 


■ 20. : Yes No DNK 


'•• 32. ; ' , Yes No DNK . . 


.9'. / Ho ' DNK ' 


21. Yes No DNK 


33. v Yes No DNK 


io. ':>;,,;^ . 


22. Ye* No DNK 


34. Yes No DNK 


11. * tei N6 i)HK 

■• .• • ■ :• • . . t ", 


23. Yes No DNK 


35. Yes No. DNK 


:. 12. :•■ Yes No r^NK 

- ■ — : — : — : .;,::..]..,. 1 


24. y Yes -No DNK 





Or 
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NTE Core Battery Teat Content Description 



Communication Skills: READISG 



Is 



II. 



III. 



topic* 



on 



Tfre ability to understand accurately ai|d COTipletely 
the explicit content of a . writ ten -message •:, • v ; 

1. ■ Main Idea .1:.C ^jl^jIiC^ ^'/J.^ ... - / 

2. Detail 
e.g. » Definition • word* phrase, etc* 

- Supporting Ideas 

Relationships . * ; x 

e.g., Sequence ? 
Cause and Effect 



Approximate 
Z 

V 50 



3. 
4. 

Analysis 



35 



•«.t 



The ability to clarify a written message and to .>:'. • * . 
understand how it is organized and conveys its message 



1. The writer's pu*j>ose 1 , ' \* 

2. The writer's assumptions : , r 

3. The writer's attitude or tone % ^ \ 

4. Implications of the message . • • v ^ | "I ; 
Inferences from the message *: ; -J!"- i'?"* " > 

5. . Fact vs. opinion in the message - 
§. Organization of the message 

7 # i ,• Use of language in' the message - 
8* Application of elements in the message 

Evaluation i 

The ability to make reasoned qualitative judgments about 
the nature and merits of a written f essage° 



« jj* A. v '" 



15 



U 
2. 
3. 

4; 



Motional of _n|nlpuiat?ive ispicts bf the message 
Strengths and/or weaknesses of the argument * 
Relevance and/or appropriateness of supporting 



Relation of the massage to the audience and/or to 
the general universe of ■ithe topic 
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NTE Core B^tterj^Cest. Content Description 

*. • ' ' • ■ -. . " , • ■ ' 

. v ; s Xonnmiii%tlc>n Skills : WRITING ' ,- 

' Topic * : " ^ > " * * 

Usage - including capitalization and punctuation, 
■y.^iSu^j ect^y^b^greeMnt^^erb form, pronoun problems, 
u parallelism, diction, idiom, structural problems , 

and adjective-adverb confusion . - ■ 



Sentence Correctioh - includ 



mm 



Approximate 
% S>f ' Test 



tug problems of coherence, 
Went, 



word order, econcnny of stateient^ appropriateness of 
diction and choice of _ idiom, subordination of sentence 
\ elements, logical comparison structure., and clarity of 
modification and pronoun reference 



55 



45 
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TlTE Core Battery Test Content Description 

General Knowledge: LITERATURE AND FINE ARTS 

''•>- ... ; . ;; .: * " : ; ■ > ::: ; V- . >: ..; ; . : : : • 

Tonic '-K-" r ■ '•• r- ,- > 




I. Recognizing basic elements and components of works 29 
■ ^: ^ of literature and fine arts 

II; Analyzing and interpreting works of literature and 54 



fine arts 



iili Relating works of literature and fine arts ^o one ., r \ 17 
another and to their social/historical context i'j 



AS? .. f 
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NTE Core Battery Test Content . Description 
General Knowledge: MATHEMATICS 

Topic 



Approximate 



I. Has good number sense arid understands fiov numbers 20 
$ . 'behave * " . N 



II. Understands and uses numbers in an appropriate way § ; . - 



to quantify thinking 



III. Recognizes and uses mathematical relationships " : i4 

IV. : Un^rstands the mathematical basis of measurement 16 

V. ^iiia^ / \ 12 




_ie?^el graphic » symbfciielland verbal material 12 



GopyrigHti 0 1982 by Educational Testing Service. All rights .reserved. 
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HTE Core BatterjrTesj Content Description 
General Knowledge: SCIENCE 

' ■ ■■ * ■ Approximate 

; topic % of Test 



!• Demonstrates understanding of energy relationships in 11 
both living and nonliving contexts 

II* Demonstrates understanding of the sigtiif leant features 11 
of living things 

III. Demonstrites understanding of the fact that the bM^f^p* 11 
tion of natural processes has resulted in -organi^S^^ " 
that fill a vast number of ecological niches and 
these organisms tffe usually classified on a, struc^^^^ 
" ^~ v basis inters smalt number of categories 9 b which; fac^l'llkte 
the understanding and study of the organisms '/'■VV'-^'- 

IV. Demonstrates imderstanding of the relationships between 
living organisms, particularly huitians, arid the , 
■ ,: ^environment ' ... ■? "^-^ s \. ""'v.. \ 

V. r. Demonstratfes understanding of the fact ttiat.Eayth' is? bofefe 11 
a part of the Universe and a hpdy that has special 
characteristics Y-^"* ■ . . "° t 

VI. Demonstrates understanding that all matter is qchSposed^ 11 




of 4^oms t ^that atfyns are "divisible, and that atoms 
undergo combinations ; . : £ !. ' ' 

VII, * Demonstrates understanding of tlje forces thai act on-Juriits 11^ 

of matter • * : " v ,• ' V : > J *\ 

.•./"•_;■ \_. ; ;. i;-: L :-_'_\! ' 'j ' i; 

•VJII. . Demonstrates understanding of the methods of science: 41 • 

the kinds of reasoning and the organization of informa- 
tion that have contributed tq^^e development of science ^5 

- IX. Demonstrates understanding qf^he role of science in 11 
^ Securing and maintaining ^Important human »Muls 



Copyright © 1982 by Educational Testing Service, i&tt rights reserved. 
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V ; NTE Core Battery ^est Content Description 

General Knowledge; SOCIAL STUDIES ; 

r- — ' • * ; ' ? i Approximate - 

topic , ■. ."- • % of Test ."ir 

■ "f"* ■ .' ■■ ■ -' • . . ■ ■ >■***■.. . ■ 

I. Understanding the forces which have influenced the 25 
evolution and current state of human culture and v 
institutions - 

II. Understanding the behavior of individuals , of small 25 
groups, and o^ social institutions and the inter- 
relationships among individuals, groups, and social 
institutions J 



III. Recognizing both the universal features of world . : 'f-.2.5 

culture* and history * and the basic differences among ^ 
; v~ cultural and Mtional units . \ .•, .. ....... ' .1 " jf \ \ 

IV. p Possessing the essential toolsVand the balanced ' ^r^^tS' 

to analyse "and make informed ^jixd^fipilfce ■=. ." '■ ^. - ^ 



about society" - rv '■"'-""^■^ 




I The topic areas above will be related to the following 
major U.S. historical and cultural events and mbvraehts;. pblft^ 
and political values; prominent characteristics of u societies ;Md^^^^^^V'g||^ 
patterns of ^social change .political organixationftv political yaiu^^^^^0^^Si:0 
ship between culture and individual (e.g., processes and ^ pa 1 1 e rns of , pr e j Ua ice v'^^^r ' 
stereotyping, and discrimination); economic concepts and' processes; gedgraphical v 
features' and characteristics of human settlement and culture; and methodologies » TT 
methodological tools, and resource! of social sciences. 

« • V ' y \ ' 

. ; . . ■-, .^:&. ( y ■ . 

< - -a ™: ■ . -1 » ■ 
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NTE Core Battery Test Content Description 
(OFESSIONAL KNOWLEDGE . 



146 



v •' I. 
II. 

in. 



Topic 



f *■ 



Planning objectives, diagnosing needs, identifying 
resources V and designing instStction 



Inpienenting conditions that facilitate learning 
and instructional dia*±8&^-^;C^fe ■"" ' f : t '.. ;';••„ v : 

Eyaliteting^^ 

effectiveness and using evaluation data/ to refine^ 
instruction 



IV, Recognizing students ? constitutional rights an^ = 
state, federal*, and^ Judicial policy i and thieir * [ 
implications for classroom practice * 

V. Recognizing extra»clas^^om influences on teachers, 
and students p; including school policy ^ coi^nity 
• expectations, the media, and children Vsdeveibpmentai 
. .patterns ■ ..... - . 0 ' ■ ' 

vr. Demonstrating knowledge of the teaching profession 
and pf professional teaching behaviors 



X of Test 

24 

" 17 - 



11 



14 



mi: 



■ ^ 
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NTE Core Battery Test Content; Description 

•• ■ •» " . • ».-••'• . 1 - , .'' 

Coiimf^ LISTENING 

I. • Basic Comprehension of Message . 

message", uncez 



^?^!|l^'ccmnptat ionp /of words, an<l summarizing major 



idea) 



itm Analysis ^>£ 



Message 



' ^includes identifying assumptions, drawing 
inferences, recognizing implications, and 
identifying speaker 9 8 tone) . • 

III* Evaluation of Message ,> • • •■ ^ ^ 5 

(includes identifying and evaluating logical 
structure, assessing -appropriateness and 
effectiveness of supporting material, and 
; evaluating effect $f speaker^ tone on an 
audience) "■ V" ^V^.'- 



k Approximate 
f % of Test 



37 



30 



18 



IV • Feedback-Response 



15 



questions or dialogues) 



responses to 



ERIC 
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INSTRUCTIONS ° 

V . / ; CONTENT REVIEW FORK; 

Tfce following chart lists the major content topics that are covered in this NTE 
' testi along with the relative weight given to each topic. Compare the relative 
' emphasis of the topics listed to the relative emphasis of these topics within 
the broad ^purriculum outline of the teacher, education sequence at your 
institution. ■.. « * 

Circle the "S^ iieitr to each topic that is given about- the same e^asia lti the__ 
teacher education curriculum at your institution as in) the test. Disregard small 
percentage differences* i.e. •differences of SZ or less/ 

. • "J; ' o : " r ""' : . A . * * ■" _ . - - ■ . . : •• 

Circle the "M" next to each topic that is given, more emphasis in your institu- 
j tion 1 8 teacher education curriculum than in the test. Mark this column only if 
\in your opinion the difference is greater than 5Z. 

Circle tife lf L" next to each topic that is given lek^ emphasis" XfoW^pifejto's 1 ' 
tion 9 s teacher education curriculum than in thg test. Ttfti^&6l|&^Dniy if 

in your opinion the difference is smaller than 52. 



< 



^ ■ 



\ 
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CONTENT REVIEW FORM 
PROFESSIONAL KNOWLEDGE 



Judge's Name 



^Content Topic* i. 
PROFESSIONAL KNOWLeSe^ 


Z of test 


. f ■ :. • . : . ' 

Emphasis in the Curriculum 


. S AME 


MORE; 


LESS , 


- '. I 


24 


S 


, M \ 
■ " 


L'-t, * '■ ■ , 

■: " '.V-- • ' ' 


■ * ,'■ ' II 


25 




' - M 


, L 

. " : . ■ 






*." s : 




'!:;K ; t r - 


■ ■ ;• i^' IV- " • 


9 


,; ; v . S /. 


M 


L ; 


- — ' - ,-: ,^v,, 

. V . 


J ■ 


s ■ ' 


M 


L . , ' . 


VI 


' 14 


s 


^ M 


L 



* Refer to the accompanying Test Content Description as you complete this form 



I 
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' l^- TEST CONTENT OMISSION FORM 

_ v _ ■ _■ ■ . j 

~ " Judge's Name •' 



• 'Br 

•r 



Use the space below to list any major, content areas in your institution's 
teacher education sequence that do not appear to be covered in the Test Content fm> 
Description. Please note that the topics youi write in should be- of approxf- 
mately the level of specif icity as those listed. Do not iist topics that may A 
be subsumed under one of the categories listed « if you are uncertain whether ' 
to list a topic here, please refer back to the Test Ccmtent Description before 
doing so. ~ 



-I- 




,1 



9 

ERIC 



; i 



j" 



A 




j. / f52 >>-> 



<T Judge 9 s Name 



- '\ Based bri the general content t^ics^c^ered/ 

±n this test, select the option below that mpit closely characterize^ your 
• ^dgment regardii^ ^hfe' similarity betyeen this NTE test and the teacher Educa- 
tion sequence #tf^bur institution*^ Indicate your answer by placing an X next 

- to the response that you have chosen, then use the space below to, add any 
additional comments you may have* - •? 



1 —— fflg ^est content topics parallel the teacher education sequence at our 
. \ institution very closely # , r V .■ ^ * S ^ 

. There a;re some differences between' the ^e^t^fcontent topics and the 

teacher education sequence at bur institution, but these differences do 
not appear to be appreciable*; ^ 

There appear to be some appreciable differences between the ^est content 
* ■' topics and the teacher, education* sequence at our institution i° 



There is little, similarity betwe^ content topics an^ thi^eacher 



education sequence at our institution. X 



4- 
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/ ITEM ESTD1AT10N FORK 
Profattlonal Knovltdgt: SECTION 1 



.ludga'a Haaa 



Fors 




Dirtctiont for fating Itaa laporianea 

Rud tach tttt ittn and tha anavtr edicts which accompany It car a fully. At you raid the itti, think of the 
impor ttnet of^tha knovltdgt rtquirtd to antvtr tht itca corractly. In your opinion, dots thii itta tttt 
aaaantial feowltogt ii^ortant knovltdgt (I), or not vary inportant knowladgt (K)? Circlt tht E» I. or K 
in tht column labtlltd H Ittm Importance" to rafltct your Judgment. 

. % ■■ Diractiona for Siting Itt» Difficulty 1 » 

Aa you raad ttch tlmi think of how tha minimally knovltdgttbla ttudtnt would jitrf on cm thii it«. In^your 
opinion, wdbld the minimally knowledgtabla attodant find tfala to ha an taay itam (E). a modtrataly difficult 
item CM) • or a hard (H) it«n? Circlt tht E. M, or H in tht column labelled "Item Difficulty" to rtfltct : 
your judgment. * ■ ' * % 

= • ^ 4 \ ' ' :■:'[ • : ■ , • t. 

(Vheri you have ratad 111 itema In both queition ieti, go on to tha Taat Satiation Form.) 



Item 
Numbtr 



Item 
Impbrtanea 



1 Item 
Difficulty 



. Item . kk$H • Itatf * 
Busbar . v.Sp^,:' Importance * 



Itim 
Difficulty. 



H 



^20. 



"XjL 




21.: 



22. 



23. 



24. 



7», 



— <*>. 
I 



10. 

11. 

12. 
13. 
U. 
13. 
16. 
17. 
fl8v 



28. 
29. 
30. 
31. 
32. 



V t 
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OVERVIEW V 0F TASKS TO .BE TERFOKlSl BY MEMBERS 
: ' J - OF KNOWLEDGE' ESTIMATION- PANELS - . ' < 



The study lil yhi&h you have Iseen asked to participate is fc^Ltig conducted by' £he 
iioufsiana St^t^ 'Department of * Education (LSDE) . The purpose of the study .is to* 
■'dys:^atB^tiiB--'cdnte^\6f the National Teacher .Examinations (NTE). in relation to 



Louisiana^ teaqher. education /programs and to estimate the test- perfo'fmarice of 
vt^iw^yy -1fr»ntgigHggq;h1 p. r^pnH^Sa^gg for certification as teachers in the public 
sfchools in Louisiana./ .* - 



A; Knowledge Estimation Panel has be^n.;: established for each' section *of .the NTE/ 
. Gore Battery Tests. -You have be eaise lect"6d O to serve oh the Knowledge Estimation 
Panel fo^, \ • - ' • ' • * '-'"V ' : 

As a member^! th& Knowledge Estimation', Panel, you will be ^gj^ed to* make judg'- 
menfs about difficulty of individual test: questions f pr persons* who v have tlje 
minimum amount of knowledge necessary to complete jttie teacher education program 
required foiPriferitlf ication in Louisiana, and tbjfeeach effectively.. *You will 
also be a^ked tcr/iiake judgments aboup the imp6rtag|fe^ ^ ; 
to answer each" ttest question, correctly, and to estimate the number of test items 1 
wltich fall into .each category of difficulty and 4a&portaiice. Your judgments' will 
be combined with judgments macfe by other faculty members to derive an estimate 
of the probable test performance of this group of persons. 




pAs. indicated in the NTE Design . Summary , a nutaber of faculty members will, be 
assembled in order to make their judgments. There, Wiil^ Be two meetings of the 
Knowledge Estimation Panel: a. Training Session on 0ct°obe^l4, 1982 , to provide a 
detailed* group orientation to and exercise in the,- Knowledge Estima 
procedures-; and the actuafl gating Session on October 15/ ••1982. . AtV^^^la 
time, judgments., will be ihade individually and independently; membersof . the 
panel will not confe^as^a grfeu^ nor will any member be" informed ! of 
judgments made by a«y othex; individ$a3^meipiber . The judgm^n^s 6i ail member 
a panel will be combined statistically '. by £he v LS0E to arrive at a .summary 
judgment | or the panel about each test. . The 9 summary judgments* .for : tj&e" fce£ts 
also^ will .be combined; 2nd the final results will be -published in a report 
describing the .study and' its^f ind'ings or •cbnciusibns. ' - •". -> : 



The information^ in, thi^&^^.ing^iaj iitten^ed to help yoa to prepare :jEor your 
rasks. If in studying ^ ^ne^fet.erials .you find that you- have questioit^^out the 
tasks, be teure . that they are answered during the Training Sessitffe- on October -14, 
The Test Content D^qpriAtion^ identifies ^the\ major jjjro^s bf^^^ics that are 
covered by the 1 test fcnd* indicates the illative emphasi^that is jpi^eu to each. , 
It will, serve to familiarize you*vitb the general content of -'the test before you 
seevtlie test questions themselves., / t % . > - 



YouV'have beer^. asked to pa^jpcfpate ^n- ^this^ 
the cufriculupi^at ,your in^H-tuttoQ . in 'the'Mieia(i 





ERIC 



because ybu are^fia: 
covem by' • th 

with .which ydu^illf be working 1 Bef orV ^tt^iing fne parieHT session r you 
to draw upoti^.uocal sot^rces^ of information regarding >the currictilum at .your 
itn's t i t^i^^i. such as ^ur o>34egg r H^t^Dgi spec^ curriculum planning § 

qp othe^^Srees, avaBaSle^a you. : * You ma^ aisd find it h^l^f ul ^to ^talk ^th 
cbileagues rwho^*have xaught specific courses 'that -you have nbf b^en\called| r ^pon 
to teach/. J* ' : ' ■ *"■"■- : 9'*" « r -. > ' / - 4% ; 



■4 



nation 
fy. want'.. 



4, 
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cbntributip^ to^^| %£u^' ,.aii<i_ y o^^iij^ir ^ca_tjt.o'flp^co • _par t ictpate ^re - an 

order that^he f fnai; ref^rt :p£^he r 
study/s f i^dinss ' acid" 'ioncl^sibjis $e- ~ : As : fcf oifmatige as pofsibl'e tQ ;j^6^;\whB?iiay.-'""" 
wi?h £o use it, we will ask f your petiilissi^ to identify ybti "lit the final 
report; Your individual judgments ^wiir not* be Identified.,. 



r 



When you go to" the meeting, please take this packet oftmateriais with yon. 



We very much appreciate your willingness icb participate in this important study. 



-Robert J. fi&rvue 




Chaii 

_NTE ;1 VH^d^tibn^Task Force 

4 jiana State Department of Education i- • 
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^Knowledgeable Teacher Cafldidata 




For the purpose of this study, use a ;ft|j^f of refgrepce the qfnimai ama^^f " 
academic knowledge needed, ,progranr require* .fdr, 

certification in 'the ^tate* .^^SSf^^i^^^'- Aif^Pselsiijent ofVninpial- || 
i is a task^that every <acjflty tteaj^r^ performs, at least" in part, every 



tlm6 hfc °r she writes and grades >xaminatiqris for * students enrolled in college* ' 
jourses oj evaluates a student tea ctier^g >perfoi^nce^ the dividing lirie*between ; 
a' minimally passing and^ failing gride must be established, by a faculty mitobei \ 
in desigjnin| ,ah examinat^n| Jb as got to St .S $ifiwity';.8f"i|B ' ':X'h 

questions Jo high |hat p excfpdesv student f|oin ' 



demonstrating the level of knowledge J^ o^ she has. ^hfirt, div Wing line must be 
assesse^fgain. in grading ^x«qjinaei6^s ^jjy^^ecause the "deliver of % fad 
grade ha \& such important consequences^ $^t_^ mlmberl pay 

• great attention to their cohcfe||ldn of: w ffift "thi irtnl^l^teowledgeab\e studeftt 
s^ibuld tenable to dp ^o achieve- a^passing grade t^eir examinations. V^tfil^ 
n ^^^^ 8 ^4, t° dSw upon* t^s' experience in apply teg * your conception of >iBe^' 




Lly knowledgeable? 'student tc teat spi! 



fe wtitten for tfie NTE. 
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APPENDIX rV-D 



ifiSTRtJClf.lONS FQ^kNOWLEDGE FORMS AND 
KNOWLEDGE E^f IMA%ON TRAINING EXERCISE 

'i y v 




INSTRUCTIONS 



KNOWLEDGE ESTIMATION FORMS 



161* * 
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, lour task is to make Judgjnents ; about the difficulty and importance of individual 
— el quest ions for 3 minimally knowiedgealiip persons in all teacher e*Scatibh 
* fields. You will be asjked to draw upon your experience to construct a 
^ hypothetical group pf person^ each of wh^m* in your judgment, has .the mfiriimum 



b complete the teachers education .program r^s^ired 
, and has the "m^pimum amount of knowledge to teach 
' at foxx apply, should be appropriate* to^botfc ; , 
academic' prpgram, and then using the 
^gained 'from the acadojnic $t-8gram to .teach . In i; d'rawing upon ydur^^jS^erience^, ybu * 



amouulj^bf academic Khowle 
for>t^rtif ication iii| Louisiana , 
effectively. The' spanl 
criteriaV'that is , ^omjfle| 



lards 



wi 11- probably find three types, of persons, for whom diffei 



levels^ of 5 



achievement would represent minimum knowledge: (1) those who will pursue' 
non teaching c5re4g| .after .graduation;- (2)* those whowjll pursue eiemeat^tgr ^ 
secondary J^^hing ^ careers f and (3 J those whq will ^u^^^gr^uape study. 1 This 



study is. /concerned only with-p^jpsaps who will. pursti^^feTaentary or. seeonda 
teachingve^reei^ an3 the £ssessments that you make witn ; respect to minimally 
knowledgeable persons *s 

jud'gm 



be made' with only, this .category of per son^^tf ,. mind. 

Xoiiv gagmen ts abput the test Muestion? v are^ to oe mafe^ith reference v to , your 
ej>fc^^ of .miniT^lly ^^ovyledgeab le , J^tudents , Jas descried : in the 

• ^eceding~~p5^ ....... As you\rad,each test qi^tion^ and.its underlined 4 answer , 

f ^thirik of this , group.' \Th^a ^jKig^ whether the 'minimally- knowledgeable students 
: ;Votild f ind\each^ item to be easy<E) , .moderately dif f iculfe(M)^ ^pr hard(H) / ..Circ^|. 
^thi E, M,*,,or H itj^the columnf labelled "item" Difficulty" on' the Item EsHiat£bfi^ 
£ F^tin , to reflect 1 ; #o^r judgtiient . Now, think b£ the_imp6rtanc^^f the " " 



to 



idge^ 
;thS It* 



Dwledge 



Lrclfc the/E, 



iry ^ important 



rtarice 



tent 




the knowledge 

nswer'the Sfem .cprrectiy. In fhxii Ifiinion,- ddfei- fchis^lit'emi jtest 
' e; ;(|r i5ppr£ant© knowledge ^5) ,^ ; or^fecfe^ : ' , 
j- ^pr^N i^>^ie' cojiimri labelled "1 
ref lfect^^?f>Jfudgmerit. *i. , H ^ . - s 

you with hp b^gi^ wl^tsQ^fe ^gjor 

may choose not to' rate th^^^|m. Do 
f ficulty in making a judgmeil^ ^you are^ 
a difficult-one. You wo^ld ^hdose; ng)^ J^r rate 
sis for making any- judgmer 



lat your ^experience 
_ ioyt J one of the qjies 

>^n& islcip/item^ simply b(ecaps^ybu 
j^-to'w&efa decisibh fvfe- 
an Htram "only when, yo; 




ng your Judgmen^sy^pu are not to be conce™ed>^out how man^gues 



^^e a'ssigr 
four h^st juc 



parlous Categories \ It x^^ur.^responsibility to 
^aiuatiSg^ dach question indi'vrida^^y. 



oii fcave 
ues^on 
$s pbfisistently Ihrj 

The ^p^eftf^rs^ „wfio >&11 be 
timber ^2. 'f)e^icil , W • eras! 
iaha| ^tidgment abag| each s 

' " '4.' 
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3 udgmeirts about the quqs tion^ __- in tl 
makec^iuare that you have ^|^ed yb| 
e set. 

r*\ . , 

ill appreciai^Jemir tfsing a 
e your^ miiid^so that your 
hdicated.: >^ ^ \ ' 




5>*.-- 



As you feur^i to :each hew test item/ 4>lea§iP be sure • that; the number ;.bf f 
quest ipn. z 6n"'^e ^age corresponds . bq : 'tfi^ Jnfimber on . the Item* Es titrat ion"' "^e^i 
^hen you^ §ave made f judgment^ "about 'ail the; questionl^in /the jqtfe s tibn .set pch^<^| 
that, the number of tfre last; Iques^l^ 
the' form corresponds, to the number -of 't^ 



Atter you' have rated al^-' items / in-, both question sets* .you. will be -asked to 
'xompletej a second form. Review the. decisions ybu have mad^aBdut the ratings. Jeff 
; itejn difficulty ana importance* Consider the itejas ytS^n^y elated as" eja s y *" ^andtC 
- "essential. In ; "y our opinion, what percent of these items would; be an^wer^d. p 

correctly by the. minimally knowledgeable student? . In other^ words, if . there ^ere 
. 100 easy/essential items, hew manv would the minimally knowledgeable student be 
able to answer correctly? Make j|a similar judgment for each 'of the" ^•P^ 
combinations of difficulty arid iftpfeftarice; Record your judgments in the spaces 
provided on the Test Estimation Form. ' ' ':■ 
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KNOWLEDGE ESTIMATION TRAINING EXERCISE-:. 

_ : - , • _ _f| _ • 

Directions for Rating Itfem Difficulty fe-'^^l 

j ^efti a nd the answer choices* wriich accompany ^it carefully v As you 
^^H^^gL£ em^ , r7h ink_qf how the minimally knowledgeable student;^#uld per^ 
f briii bit this it em . In your opinion, . would the minimally ^fcnowledgfable '.stu- 
" /dent find this^to te'Sn east item (E) , a moderately difficult 'item (M) , or a 
- hard * item- (H>?» Circl^the E, M, ojr H in the column labelled "item Diffi- 
culty" to reflect your judgment , - ' . : 

. ■ ■ . . * • . . * ' • ' "'"' : ' V 

' . * * * Directions for Rating » Item -Importance ^S^ 





As you read each itlm, think of the import^ce of the fiiowlfedge" required to. 
answer the item ^ott^cXIy^. In your opinion, does tljis item test .essential 
knowledge (E) > important knowledge (I)* or not. very important knowledge (Nj? 
Circle the E, ' I , or N in the column labelled "item Importance" to reflect 
your judgment. *-> s 7 



.-.'■X ■ ■■*"-£r.l - • " >.< • * 
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>> : TEST ESTIMATION f DRM ■ 5 . '• 

*&r. • 



Test eoiitent 



Directions. 




When you have finished rating bath questibii 'gets" for a test , compete this form 
..r for that test. Be sure to complete the blanks at^ the' top of, this" form,, Then, 
f review the decisions you have made about the ratings of item* difficulty and\ 
impdrtaiifce. Consider the items you have rated as 'easy . and essential. In your 
opinion, wfiat percent of these items would-be answered correctly by the mini-,; . 
^-v? ^ owled S eablestude nt? Make -a^imilaf judgment for,, each of the nine J 
cognations of difficulty and importance; Record yoii^ judgments in the spaces 
below. - * - ■ ■* » '. 



v : 



DIFFICULTY 



IMPORTANCE" 



$ OF ITEMS 



v. 



* 



r 




Enter tfie : .yinmS^r ^of 




ion Tfcaigifig Worksheet ■ . 

as Easy/E ss entia i , Easy/ Important , - 



etc, iijto each cell of ..the 

2. Multiply each of these ;^^|^ - by ' 1 0 , and enter : • t h is H*k number on the 
next, line in each cell of t^^ftiltfix- > The purpose! of this Multiplication is 
to iicrease_ ar^ in each cell while maintaining 

thfe same proportions as the* origin! 1 rated items 1 . r| , ' '- V 

■4j3i Enter the percentages of items at each level of difficulty and import^pe 
/Which ttfie minimally knowledgeable student should ^answer^ correctly., into each 
, cell of the matrix ^ on. the liiie marked "% n . These are the percentages you 
'entered on athe Test Estimation FbrSV \. ' y : $ . >vj 

• • .• ;\ \ \ ■ - . ' ; i° ' ' " '• •• • ' S ■* .- 

4* Multiply, this percentage by » the number 'in the "items X 10" space; enter . 
the '* product in 1 the space; markcj^ "Correct 1 % _ -Foi\ example if ther^ are 20 
: items in the "items X : 10" ispacl^id. 90 in' the 'ft' 1 space; the number in the 
"Correct" * space: would be 20 3C *:90 = 18/ Perform this ^tabulation* 5 for^each 
: ^ll^in the matrix. • • % ... J&$% ./ .'■ V v " '• ■ •', 




ft' 



v % 



Item 



ultv 



••- EASY 



MODERATE 1 *> 



KARD 



: #|SSENTI&L 





^ 1_ (C j , 

* - "Items : " v . 
■ • ■ Items, X 10 

' COBREST . 


Jtems^_ y 

Iterjis X^IO 

% '■ 

CORRECT i 


■ ' Items v.: > ' 
"3.€*BR^CT^. 


.Items y _■_ 
"! 'J Items X 10 

• . . x-- - •• • ■ ■ 
• .CORRECT 

> . ' I - 


' Item^ , 
; Items rX 10. 

i CORRECT : 

» i' ■? * » * 


Items 

. Items X 10 U 

■ : ■ • m 

CORRECT 

- . - ' v. ' 


It^ms/ 
— Items X 10 

... CORRECPs 


•jff- - 

Items 

Items X 10 

'II. t 

CORRECT , 


* ' ' Items 

Items X 10 \ 

; CORRECT 

* '-■ • . - 
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